English

Reinforcement learning is the discipline of learning to act through interaction. Unlike supervised learning, which trains on labeled examples of desired input-output behavior, or unsupervised learning, which finds structure in unlabeled data, reinforcement learning involves an agent taking actions in an environment, observing the consequences, and adjusting its behavior to maximize accumulated reward over time. The agent must discover what to do not by being told but by trying things and observing results — which creates distinctive challenges that supervised learning does not encounter.

Three problems characterize the discipline. The credit assignment problem: when a reward arrives, which of the many preceding actions caused it? An agent that plays chess for fifty moves and then wins must somehow determine which moves mattered. The exploration-exploitation tradeoff: should the agent take the action that has worked best so far, or try something less certain whose value is unknown? A system that only exploits current knowledge cannot discover that better strategies exist. The sample efficiency problem: learning by interaction is expensive — real robots, medical treatments, and financial decisions cannot be subjected to millions of trial-and-error episodes. Each problem has generated its own mathematical machinery and algorithmic responses.

The subject has a formal mathematical foundation in Markov decision processes (MDPs) that distinguishes it from most of machine learning. This foundation connects RL to dynamic programming, optimal control, game theory, and decision theory, making it a genuinely cross-disciplinary subject. It also means that RL has rigorous theoretical results — convergence guarantees for Q-learning, policy gradient theorems, regret bounds for bandit problems — alongside the empirical phenomena that characterize contemporary deep RL.

Prerequisites: Machine learning foundations (§5.2) — probability theory, optimization, evaluation methodology. Deep learning (§5.3) for deep RL specifically. Algorithms (§2.6) — dynamic programming is the mathematical foundation of RL.

From Bellman’s Equations to Language Model Alignment

Reinforcement learning has a cleaner intellectual lineage than most machine learning subjects, because its mathematical foundation was formalized before its computational applications were explored.

Richard Bellman, at RAND Corporation in 1957, introduced dynamic programming as a method for solving sequential decision problems: rather than planning from the beginning to the end, decompose the problem by working backward from the end. The Bellman equation expresses the optimal value of a state — the maximum expected reward achievable from that state under optimal future behavior — in terms of the optimal values of successor states. This recursive structure is the mathematical core of reinforcement learning. The terminology “curse of dimensionality” was Bellman’s coinage for the observation that the computational cost of dynamic programming grows exponentially with the number of state variables — the fundamental obstacle that makes exact dynamic programming intractable for most realistic problems.

Arthur Samuel’s checkers-playing program (1959) was the first demonstration that a machine could improve at a task by playing against itself, without being told what to do. Samuel called this “machine learning,” and the program learned an evaluation function that let it beat an amateur champion by 1962. Crucially, Samuel’s program was not solving a problem in Bellman’s sense — it was using a cruder form of self-play to adjust its evaluation function. The formal connection between dynamic programming and learning from interaction would take decades to formalize.

The formal foundation of temporal-difference learning was laid by Rich Sutton, first in his 1988 paper “Learning to Predict by the Methods of Temporal Differences.” TD learning is the insight that a learner can improve its estimates not just by comparing predictions to actual outcomes, but by comparing predictions to later predictions. If you predict that a position is worth 0.7, and later predict it is worth 0.9, the earlier prediction can be updated toward 0.9 immediately — without waiting for the game to end. This bootstrapping — using estimates to improve estimates — is what makes TD learning more data-efficient than Monte Carlo methods that wait for full episode outcomes. TD learning is the algorithmic foundation of most RL methods.

Chris Watkins’s Q-learning (1989, thesis) made the connection to control explicit. The Q-function Q(s, a) represents the expected total future reward of taking action a in state s and then acting optimally thereafter. Watkins proved that an algorithm that updates Q-values based on observed transitions would converge to the optimal Q-function in the limit, for any finite MDP, given sufficient exploration. This was the first off-policy convergence guarantee in RL: the algorithm could learn from experience under any behavior policy, not only from experience under the current policy. Q-learning is still among the most widely taught and used RL algorithms thirty-five years after its introduction.

The connection between RL and neuroscience arrived through the discovery that dopamine signals in the brain appeared to implement something like TD learning. Schultz, Dayan, and Montague (1997) showed that dopaminergic neurons fired in response to unexpected rewards — and, crucially, that once a predictor of reward was learned, the firing shifted from the time of the reward to the time of the predictor. This was precisely the TD error signal that Sutton’s algorithm computed. The discovery was not merely analogical — it suggested that biological reward learning and computational RL might implement the same mathematical principle. This connection gave RL a privileged status among ML approaches in the cognitive neuroscience community and contributed to the field’s prestige.

The deep learning era transformed RL from a subject with elegant theory and limited application to a source of dramatic empirical results. In 2013, Mnih et al. at DeepMind trained a single system — the Deep Q-Network (DQN) — to play 49 Atari games at superhuman level, using only raw pixel inputs and the game score as reward. The same architecture, without game-specific features, learned to play all 49 games. The combination of Q-learning with deep convolutional networks demonstrated that the theoretical RL framework could be instantiated at the scale of complex perceptual input, a combination that classical RL had not achieved.

AlphaGo’s 2016 victory over Lee Sedol was the most publicly visible RL success, but its architecture was a hybrid: Monte Carlo Tree Search for planning, supervised learning on human games for initial policy training, and RL for self-play refinement. AlphaZero (2017) replaced the human game data with self-play from random initialization and achieved similar results, demonstrating that the combination of MCTS and RL could discover strong strategies in large game spaces without human supervision. MuZero (2019) removed the need for perfect environment rules by learning a model of the environment alongside the policy, extending the approach to settings where the dynamics are not directly given.

Policy gradient methods, which optimize policy parameters directly by estimating the gradient of expected reward, emerged as a distinct and important algorithmic family. Williams’s REINFORCE algorithm (1992) provided the theoretical basis. The challenge was variance: the gradient estimate was unbiased but had very high variance, making learning slow and unstable. Actor-critic methods (which estimate a value function to reduce variance in the gradient estimate) and trust region methods (Schulman et al.’s TRPO, 2015) addressed this. Proximal Policy Optimization (PPO, Schulman et al., 2017) provided a practical, stable policy gradient algorithm that became the most widely used RL algorithm in the deep RL era — a position it retains because of its reliability across diverse settings.

The application of RL to language model alignment came through Reinforcement Learning from Human Feedback (RLHF). The central challenge of aligning language models to human preferences is that the desired behavior — helpfulness, harmlessness, honesty — cannot be easily specified as a formal reward function. RLHF sidesteps this by learning a reward model from human comparisons of model outputs, then using RL to fine-tune the language model against the learned reward. Christiano et al.’s 2017 paper “Deep Reinforcement Learning from Human Preferences” demonstrated the approach in simpler settings. The InstructGPT paper (Ouyang et al., 2022) showed that RLHF could substantially improve the alignment of large language models, producing models that followed instructions more reliably and were rated more helpful and less harmful by human evaluators. The technique was adopted across the industry: GPT-4, Claude, Gemini, and most deployed frontier models use RLHF or variants of it. Direct Preference Optimization (DPO, Rafailov et al., 2023) later showed that the RL step in RLHF could be replaced by a supervised objective that directly minimizes a preference-based loss, simplifying the procedure.

The contemporary RL landscape is broader than the deep RL narrative suggests. Offline RL — learning policies from fixed datasets without online interaction — has become important in settings where environment interaction is expensive or unsafe. Model-based RL, which learns a model of the environment and plans using it, has seen renewed interest as a path to better sample efficiency. Multi-agent RL, where multiple agents interact in shared environments, has produced notable results in competitive games (StarCraft, Dota 2) and in cooperative settings (multi-robot coordination, traffic management). And RLHF’s industrial success has brought unprecedented attention and resources to the field, simultaneously advancing capabilities and raising urgent questions about what the resulting systems are actually learning.

The MDP Framework, Value Functions, and Policy Learning

Markov Decision Processes: The Formal Foundation

A Markov decision process is defined by: a set of states S, a set of actions A, a transition function P(s’

s, a) giving the probability of reaching state s’ from state s by taking action a, a reward function R(s, a) giving the immediate reward for taking action a in state s, and a discount factor γ ∈ [0, 1) that reduces the value of future rewards relative to immediate ones. A policy π(a

s) specifies the probability of taking action a in state s. The goal is to find the policy that maximizes expected discounted cumulative reward.

The Bellman equations express optimal behavior recursively. The optimal value function V(s) satisfies V(s) = max_a [R(s,a) + γ Σ_{s’} P(s’

s,a) V(s’)], stating that the optimal value of a state equals the immediate reward plus the discounted optimal value of the successor state, maximized over actions. The optimal Q-function satisfies Q(s,a) = R(s,a) + γ Σ_{s’} P(s’

s,a) max_{a’} Q*(s’, a’). These equations have a unique fixed-point solution, and iterative methods (value iteration, policy iteration) compute it in finite MDPs.

The Markov property — that the next state depends only on the current state and action, not on the history — is both the framework’s strength and its limitation. It enables the recursive Bellman structure that makes the mathematics tractable. It excludes settings where partial observability requires maintaining history (POMDPs), settings with non-Markovian reward functions, and settings where the state space must be inferred from observations. Extensions of the MDP framework address these, at the cost of additional complexity.

The discount factor γ governs the tradeoff between immediate and future rewards. At γ = 0, only immediate rewards matter; the agent acts myopically. At γ approaching 1, future rewards are nearly as valuable as immediate ones; the agent acts far-sightedly. The discount factor also determines whether the total reward sum converges, and higher γ makes the value function more sensitive to long-term consequences, which can make learning harder.

Value-Based and Policy-Based Methods

Q-learning is the canonical value-based method: maintain an estimate Q(s, a) of the optimal Q-function, update it toward bootstrapped targets when transitions are observed, and act greedily with respect to the current estimate (with ε-greedy exploration). DQN uses a deep neural network to represent Q, with two specific stabilizations: a replay buffer that stores and randomly samples past transitions (breaking the correlation in consecutive experience), and a target network that holds the bootstrap target fixed for many steps (preventing the instability of chasing a moving target).

Policy gradient methods directly parameterize the policy and optimize the expected cumulative reward by gradient ascent. The policy gradient theorem (Sutton et al., 2000) gives the gradient of expected reward with respect to policy parameters: ∇J(θ) = E_{τ~π_θ}[Σ_t ∇log π_θ(a_t

s_t) A_t] where A_t is the advantage — the difference between the Q-value of the action taken and the baseline value of the state. Actor-critic methods learn both a policy (actor) and a value function (critic), using the critic to estimate advantages and reduce variance.

The on-policy/off-policy distinction is fundamental. On-policy methods (SARSA, PPO) learn about the current policy from data generated by that policy. Off-policy methods (Q-learning, SAC) can learn about one policy from data generated by a different policy. Off-policy learning allows data reuse — storing experience in a replay buffer and sampling from it many times — which is crucial for sample efficiency. The theoretical stability of off-policy learning with function approximation is more fragile: the “deadly triad” of function approximation, off-policy learning, and bootstrapping can produce divergence, and several important algorithms failed in practice because of this.

PPO’s widespread adoption stems from its balance of simplicity, stability, and performance. It constrains each policy update to avoid large steps — measured by the ratio between the new and old policy’s probabilities — which prevents the catastrophic performance degradation that unconstrained policy gradient steps can cause. The implementation is simpler than TRPO’s second-order constraint while achieving comparable stability.

Credit Assignment, Exploration, and Sample Efficiency

Temporal-difference learning addresses credit assignment by decomposing the long-term problem into overlapping short-term problems. Each step produces a TD error — the difference between the bootstrapped target and the current estimate — which propagates credit backward through time. Eligibility traces generalize this by assigning credit to recent actions in proportion to how recent they were, interpolating between one-step TD (which assigns credit only to the immediately preceding action) and Monte Carlo (which assigns credit uniformly to all actions in the episode).

The multi-step returns of n-step TD methods and the λ-weighted returns of TD(λ) provide additional credit assignment mechanisms. PPO and modern actor-critic methods typically use generalized advantage estimation (GAE, Schulman et al., 2016), which computes a weighted combination of n-step advantages to balance bias and variance in the gradient estimate.

Exploration in deep RL is harder than in tabular settings. The ε-greedy strategy works for small state spaces where states are revisited frequently; in large state spaces, random exploration wastes too many steps on unproductive actions. Intrinsic motivation approaches give the agent a bonus reward for visiting novel states, measured by prediction error of a learned model (curiosity-driven exploration, Pathak et al., 2017) or by disagreement among an ensemble of models (or by a fixed random network prediction error, Random Network Distillation, Burda et al., 2018). These approaches have enabled exploration in hard exploration environments like Montezuma’s Revenge where ε-greedy exploration fails completely.

What Studying This Changes

Reinforcement learning changes how practitioners think about sequential decisions and about the relationship between specification and behavior.

The first change is fluency with the MDP framework as a modeling language. The translation of a real problem into an MDP — defining what the state is, what the action space is, what the reward function should be, what the discount factor represents — is itself a significant analytical activity. Practitioners who have internalized the MDP framework approach decision problems by asking these questions systematically, rather than treating the problem structure as fixed by whatever framework happens to be available.

The second change is the disposition to think about decisions sequentially and their consequences long-term. Supervised learning asks “what is the right output for this input?” RL asks “what policy is optimal over time?” This reframing reveals how short-term and long-term objectives can conflict, why greedy policies often fail in sequential settings, and why the credit assignment problem is genuinely hard. Practitioners who have internalized this think differently about optimization objectives in any domain where actions have downstream consequences.

The third change is methodological discipline in empirical RL. Henderson et al.’s “Deep Reinforcement Learning that Matters” (2018) showed that results in deep RL papers were frequently unreproducible, with different random seeds, different hyperparameter choices, and different implementation details producing qualitatively different results. A practitioner who has studied RL knows to run experiments with multiple seeds, report variance, document implementation details, and treat published results with appropriate skepticism. This discipline is more important in RL than in most ML because RL’s variance is structurally higher.

The fourth change is appreciation for the reward specification problem as a fundamental challenge. Reward functions specify what the agent is asked to maximize — and agents maximize exactly what they are asked to maximize, not what was intended. The reward hacking literature documents cases where agents found unexpected solutions to the stated objective that violated the spirit of the reward. This is not a bug to be fixed; it is a structural property of optimization under misspecified objectives. Understanding this shapes how practitioners approach RLHF, goal specification in robotics, and any setting where an agent is given an objective to maximize.

Resources

Books and Texts

Sutton and Barto’s Reinforcement Learning: An Introduction (2nd ed., 2018, free at incompleteideas.net) is the canonical text. Written by the field’s founders, it presents RL as its designers see it — with careful mathematical development, intuition alongside formalism, and a historical perspective that clarifies where each idea came from. The second edition extends the first to include deep RL connections. Working through Sutton and Barto cover to cover, doing the exercises, is the right path for anyone who wants to understand RL rather than just use it. The freely available status removes any barrier.

For the mathematical foundations and connection to optimal control, Bertsekas’s Dynamic Programming and Optimal Control (Volumes 1 and 2, Athena Scientific) and Reinforcement Learning and Optimal Control (2019) provide rigorous treatment at a level above Sutton and Barto. Bertsekas’s perspective is that RL and optimal control are the same subject viewed from different communities, and his books demonstrate the connections explicitly.

Agarwal, Jiang, Kakade, and Sun’s Reinforcement Learning: Theory and Algorithms (free at rltheorybook.github.io) provides contemporary theoretical RL at graduate level — PAC bounds, sample complexity, the theory of function approximation stability — for learners who want theoretical rigor.

Lattimore and Szepesvári’s Bandit Algorithms (Cambridge, 2020, free at tor-lattimore.com) covers the bandit problem — the RL special case of stateless decisions — with full mathematical rigor. Bandits are both a prerequisite for understanding exploration theory and an important application domain in their own right (clinical trials, recommendation systems, A/B testing).

Book	Role	Type
Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., free)	Canonical foundational text	Entry
Bertsekas, Reinforcement Learning and Optimal Control	Optimal control connection; mathematical depth	Depth
Bertsekas, Dynamic Programming and Optimal Control (2 vols.)	DP foundations at depth	Reference
Agarwal, Jiang, Kakade & Sun, RL: Theory and Algorithms (free)	Contemporary theory	Depth
Lattimore & Szepesvári, Bandit Algorithms (free)	Bandit theory	Depth
Puterman, Markov Decision Processes	MDP mathematical foundation	Reference

Courses and Lectures

Berkeley CS285 (Deep Reinforcement Learning, Sergey Levine, free lecture videos on YouTube) is the standard graduate course on deep RL. The lectures cover the algorithmic and theoretical foundations with the depth that the contemporary literature requires. The assignments involve implementing standard deep RL algorithms on standard benchmarks, which is the right practical introduction.

David Silver’s RL lecture series (DeepMind/UCL, free on YouTube) covers the foundational RL material — MDPs, dynamic programming, TD learning, function approximation — with exceptional clarity and remains the best video introduction to the theoretical foundations.

Sutton’s online course at the University of Alberta (via Coursera, free to audit) provides structured pedagogy from the author of the textbook, with the textbook as companion.

Course	Platform	Type
Berkeley CS285 Deep RL (free)	YouTube / course site	Entry
David Silver RL lectures (free)	YouTube	Entry
Sutton’s Alberta RL course (free to audit)	Coursera	Entry
Hugging Face Deep RL Course (free)	huggingface.co/learn	Practice

Practice, Tools, and Current Sources

OpenAI Spinning Up (free at spinningup.openai.com) provides both an introduction to deep RL and clean implementations of core algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC) in PyTorch. The implementations are readable and serve as the canonical reference for how these algorithms are correctly implemented. Working through Spinning Up, implementing at least one algorithm from scratch by comparison with the reference, is the right practical entry.

Gymnasium (the maintained fork of OpenAI Gym, free at gymnasium.farama.org) provides the standard RL environment API, with classic control, Atari, MuJoCo, and other environments. Every deep RL algorithm is evaluated on Gymnasium-compatible environments; familiarity with the API is practically necessary.

Stable Baselines3 (free) provides production-quality implementations of DQN, PPO, SAC, TD3, A2C, and other standard algorithms with a consistent API. Useful for baseline comparison and for applying standard algorithms to new environments without reimplementing them.

Henderson et al.’s “Deep Reinforcement Learning that Matters” (NeurIPS 2018, free) is required reading before any serious empirical RL work. It documents the reproducibility problems in published deep RL results and specifies the methodological discipline — multiple seeds, full hyperparameter reporting, statistical significance — that reliable empirical RL requires.

Resource	Platform	Type
OpenAI Spinning Up (free)	spinningup.openai.com	Practice
Gymnasium (free)	gymnasium.farama.org	Practice
Stable Baselines3 (free)	stable-baselines3.readthedocs.io	Practice
CleanRL (free)	GitHub	Practice
RL Baselines3 Zoo (free)	rl-baselines3-zoo.readthedocs.io	Practice
Safety Gym (free)	openai.com	Auxiliary
Henderson et al., “Deep RL that Matters” (free)	Deep RL reproducibility landmark	Depth

Traps

Trap	Why it misleads	Better response
Skipping tabular RL to go straight to deep RL	The DQN paper and the impressive Atari results make deep RL the entry point for many learners, bypassing the tabular RL that provides the conceptual foundation. The result is practitioners who can instantiate DQN but cannot explain why replay buffers are needed, cannot recognize when Q-learning will diverge, and cannot adapt when deep RL fails.	Work through Sutton and Barto’s tabular chapters before deep RL. Implement Q-learning and SARSA from scratch in GridWorld-style environments. Understand the theoretical convergence results in tabular settings before accepting the empirical success of deep RL approximations as the explanation.
Treating reward design as a minor implementation detail	Specifying a reward function is specifying an objective, and agents optimize exactly what they are given. Reward hacking — where the agent achieves high reward through behaviors that violate the intent of the reward — is documented in the literature and in production RL deployments. Getting the reward wrong produces an agent that succeeds at the letter while violating the spirit.	Study the reward hacking literature before designing reward functions. Treat reward specification with the same care as API specification: what behaviors does this reward incentivize that were not intended? What behaviors could a sufficiently capable agent find to exploit this reward? Read Krakovna et al.’s specification gaming examples repository.
Ignoring the reproducibility problem	Published deep RL results frequently do not reproduce reliably across different random seeds, implementations, and hyperparameter configurations. Practitioners who do not account for this will mistake random variation for meaningful differences between algorithms.	Run experiments with at minimum 5-10 random seeds, report mean and standard deviation or confidence intervals, do not draw conclusions from single runs. Read Henderson et al. (2018) and treat its methodological recommendations as minimum standards, not optional.
Applying RL where supervised learning is appropriate	RL is appropriate when the problem has genuine sequential structure — decisions have consequences for future state — and when environment interaction is available. Many problems that look like RL problems (recommendation, classification with uncertain feedback) are better solved with supervised learning on logged data, bandit algorithms, or offline RL. Applying online RL where interaction is expensive or dangerous adds sample inefficiency and safety risk unnecessarily.	Ask whether the problem genuinely requires online interaction and sequential decision-making. If the problem can be framed as supervised learning on historical data, do that first. If the reward signal is available immediately and actions don’t affect future states, use bandit methods rather than full RL.
RLHF as solved alignment	RLHF substantially improves language model behavior compared to pretraining alone, and its success has led some practitioners to treat alignment as a problem with a known engineering solution. In practice, RLHF produces models aligned to the specific preferences expressed in the training data — which may reflect annotator biases, may reward sycophantic responses, and may not generalize to novel situations or higher capability levels.	Treat RLHF as a tool with known limitations rather than a solution. Study the documented failure modes: sycophancy (models agreeing with users rather than being accurate), mode collapse (reduced response diversity), and overoptimization (Goodhart’s law applied to the reward model). Read the DPO paper as an alternative approach. Understand that alignment research is ongoing and that current methods are partial solutions.
Overlooking sample efficiency requirements	Online deep RL often requires millions of environment interactions to learn — orders of magnitude more than humans need for comparable tasks. Applying online RL to settings where interactions are expensive (physical robots, medical treatments, financial decisions) without accounting for sample efficiency leads to systems that are theoretically capable but practically impractical.	Before applying RL to a new domain, assess the cost of environment interaction and the sample budget available. If online interaction is limited, consider offline RL from logged data, imitation learning from demonstrations, model-based RL to reduce required samples, or hybrid approaches. Sample efficiency is a design constraint, not an implementation detail.

中文

强化学习是一门研究如何通过交互来学习行动的学科。监督学习是在带标签样本上训练期望的输入—输出行为；无监督学习是在无标签数据中发现结构。与二者不同，强化学习涉及一个智能体在环境中采取行动、观察后果，并调整自身行为，以最大化随时间积累的奖励。智能体不是通过被告知答案来发现该做什么，而是通过尝试行动并观察结果来发现——这也带来了监督学习不会遇到的独特挑战。

这门学科由三个问题刻画。信用分配问题：当奖励到来时，前面许多行动中到底是哪一个造成了它？一个下了五十步棋然后获胜的智能体，必须以某种方式判断哪些走法真正重要。探索—利用权衡：智能体应该选择目前为止效果最好的行动，还是尝试一个价值未知、但可能更好的行动？一个只利用当前知识的系统，无法发现更好的策略存在。样本效率问题：通过交互学习是昂贵的——真实机器人、医疗治疗和金融决策都不可能经历数百万次试错回合。每个问题都产生了自己的数学机制和算法回应。

强化学习在马尔可夫决策过程（MDPs）中拥有形式化数学基础，这使它区别于机器学习的大多数领域。这一基础把 RL 与动态规划、最优控制、博弈论和决策理论连接起来，使它成为一个真正跨学科的主题。这也意味着，RL 同时拥有严格的理论结果——Q-learning 的收敛保证、策略梯度定理、bandit 问题的 regret bounds——以及当代深度 RL 所具有的经验现象。

前置知识：机器学习基础（§5.2）——概率论、优化、评估方法论。深度学习（§5.3）——特别用于深度 RL。算法（§2.6）——动态规划是 RL 的数学基础。

从 Bellman 方程到语言模型对齐

与机器学习中的大多数主题相比，强化学习拥有更清晰的思想谱系，因为它的数学基础在计算应用被深入探索之前就已经形式化。

Richard Bellman 于 1957 年在 RAND Corporation 提出动态规划，将其作为求解序列决策问题的方法：不要从开头一直规划到结尾，而是从终点向前反向分解问题。Bellman 方程用后继状态的最优价值来表达一个状态的最优价值——也就是从该状态出发，在未来采取最优行为时可以获得的最大期望奖励。这种递归结构是强化学习的数学核心。“维数灾难”（curse of dimensionality）这个术语也是 Bellman 创造的，用来描述动态规划的计算成本会随着状态变量数量指数级增长——这是使精确动态规划在大多数现实问题中不可行的根本障碍。

Arthur Samuel 的跳棋程序（1959）首次展示了机器可以在没有被告知该做什么的情况下，通过与自己对弈来提升任务表现。Samuel 把这称为“机器学习”，这个程序学习了一个评估函数，并在 1962 年击败了一位业余冠军。关键在于，Samuel 的程序并不是在 Bellman 的意义上求解问题——它使用的是一种更粗糙的自我对弈形式，用来调整自己的评估函数。动态规划与从交互中学习之间的形式化连接，还需要几十年才会被明确建立。

时序差分学习的形式基础由 Rich Sutton 奠定，首先体现在他 1988 年的论文 “Learning to Predict by the Methods of Temporal Differences” 中。TD 学习的洞见是：学习者不仅可以通过把预测与实际结果比较来改进估计，也可以通过把当前预测与后来的预测比较来改进估计。如果你预测某个局面价值为 0.7，后来又预测它价值为 0.9，那么早先的预测可以立刻朝 0.9 更新——不必等到整局游戏结束。这种 bootstrapping——用估计来改进估计——使 TD 学习比必须等待完整回合结果的蒙特卡洛方法更具数据效率。TD 学习是大多数 RL 方法的算法基础。

Chris Watkins 的 Q-learning（1989，博士论文）明确建立了它与控制问题的连接。Q 函数 Q(s, a) 表示在状态 s 中采取行动 a，然后之后都采取最优行动时，预期能够获得的未来总奖励。Watkins 证明，对于任何有限 MDP，只要有充分探索，一个基于观测到的转移来更新 Q 值的算法，最终会收敛到最优 Q 函数。这是 RL 中第一个离策略收敛保证：算法可以从任何行为策略下的经验中学习，而不只能从当前策略下的经验中学习。Q-learning 在提出三十五年后，仍然是最常被教授和使用的 RL 算法之一。

RL 与神经科学的连接，来自这样一个发现：大脑中的多巴胺信号似乎实现了某种类似 TD 学习的机制。Schultz、Dayan 和 Montague（1997）表明，多巴胺神经元会对意外奖励产生反应——更关键的是，一旦奖励预测信号被学会，神经元放电就会从奖励发生的时间转移到预测信号出现的时间。这正是 Sutton 算法所计算的 TD 误差信号。这个发现并不只是类比性的——它暗示生物奖励学习和计算 RL 可能实现了同一个数学原则。这种连接使 RL 在认知神经科学共同体中获得了机器学习方法中的特殊地位，也提升了这个领域的声望。

深度学习时代把 RL 从一个理论优雅但应用有限的主题，转变成了产生戏剧性经验结果的来源。2013 年，DeepMind 的 Mnih 等人训练了一个单一系统——Deep Q-Network（DQN）——只使用原始像素输入和游戏分数作为奖励，就在 49 个 Atari 游戏中达到超人水平。同一个架构，在没有游戏特定特征的情况下，学会了玩全部 49 个游戏。Q-learning 与深度卷积网络的结合表明，理论上的 RL 框架可以在复杂感知输入的规模上被实例化，而这是古典 RL 此前未能实现的组合。

AlphaGo 在 2016 年战胜李世石，是 RL 最广为公众所知的成功，但它的架构是混合式的：蒙特卡洛树搜索用于规划，监督学习在人类棋局上进行初始策略训练，RL 用于自我对弈微调。AlphaZero（2017）用从随机初始化开始的自我对弈取代了人类棋局数据，并取得类似结果，证明 MCTS 与 RL 的结合可以在巨大博弈空间中发现强策略，而不需要人类监督。MuZero（2019）进一步去除了对完美环境规则的需求：它在学习策略的同时学习环境模型，将这种方法扩展到动态规则并不直接给定的场景。

策略梯度方法直接优化策略参数，通过估计期望奖励的梯度，成为一个独立且重要的算法家族。Williams 的 REINFORCE 算法（1992）提供了理论基础。挑战在于方差：梯度估计是无偏的，但方差非常高，使学习缓慢且不稳定。Actor-critic 方法（通过估计价值函数来降低梯度估计方差）和信赖域方法（Schulman 等人的 TRPO，2015）回应了这个问题。近端策略优化（PPO，Schulman 等，2017）提供了一种实用且稳定的策略梯度算法，并成为深度 RL 时代使用最广泛的 RL 算法——它至今仍保持这一地位，因为它在多种环境下都足够可靠。

RL 在语言模型对齐中的应用，是通过基于人类反馈的强化学习（Reinforcement Learning from Human Feedback，RLHF）进入的。将语言模型对齐到人类偏好的核心挑战在于，期望行为——有帮助、无害、诚实——很难被轻易指定为形式化奖励函数。RLHF 绕过这个问题的方式是：先从人类对模型输出的比较中学习一个奖励模型，然后使用 RL 针对这个学到的奖励来微调语言模型。Christiano 等人 2017 年的论文 “Deep Reinforcement Learning from Human Preferences” 在较简单场景中展示了这种方法。InstructGPT 论文（Ouyang 等，2022）表明，RLHF 可以显著改善大语言模型的对齐，使模型更可靠地遵循指令，并且在人类评估者眼中更有帮助、更少有害。这项技术被整个行业采用：GPT-4、Claude、Gemini，以及大多数已部署的前沿模型，都使用 RLHF 或其变体。直接偏好优化（Direct Preference Optimization，DPO，Rafailov 等，2023）后来表明，RLHF 中的 RL 步骤可以被一个监督目标取代，直接最小化基于偏好的损失，从而简化流程。

当代 RL 图景比深度 RL 叙事所暗示的更广。离线 RL——从固定数据集中学习策略，而不进行在线交互——在环境交互昂贵或不安全的场景中变得重要。基于模型的 RL 会学习一个环境模型，并用它进行规划，作为提高样本效率的一条路径重新受到关注。多智能体 RL 研究多个智能体在共享环境中的互动，它在竞争性游戏（StarCraft、Dota 2）和合作性场景（多机器人协调、交通管理）中产生了显著结果。RLHF 的工业成功则给这个领域带来了前所未有的关注和资源，在推进能力的同时，也提出了紧迫问题：这些系统到底学到了什么？

MDP 框架、价值函数与策略学习

马尔可夫决策过程：形式基础

一个马尔可夫决策过程由以下部分定义：状态集合 S，动作集合 A，转移函数 P(s’

s, a)，表示在状态 s 采取动作 a 后到达状态 s’ 的概率；奖励函数 R(s, a)，表示在状态 s 采取动作 a 的即时奖励；以及折扣因子 γ ∈ [0, 1)，它会降低未来奖励相对于即时奖励的价值。策略 π(a

s) 指定在状态 s 中采取动作 a 的概率。目标是找到能够最大化期望折扣累积奖励的策略。

Bellman 方程以递归方式表达最优行为。最优价值函数 V(s) 满足 V(s) = max_a [R(s,a) + γ Σ_{s’} P(s’

s,a) V(s’)]，也就是说，一个状态的最优价值等于即时奖励加上后继状态的折扣最优价值，并在动作上取最大。最优 Q 函数满足 Q(s,a) = R(s,a) + γ Σ_{s’} P(s’

s,a) max_{a’} Q*(s’, a’)。这些方程有唯一的不动点解，迭代方法（价值迭代、策略迭代）可以在有限 MDP 中计算出它。

马尔可夫性质——下一状态只依赖当前状态和动作，而不依赖历史——既是这一框架的力量，也是它的限制。它使 Bellman 递归结构成为可能，从而让数学可处理。但它排除了那些需要保存历史的部分可观测场景（POMDPs）、具有非马尔可夫奖励函数的场景，以及必须从观测中推断状态空间的场景。MDP 框架的扩展可以处理这些问题，但代价是额外复杂度。

折扣因子 γ 控制即时奖励与未来奖励之间的权衡。当 γ = 0 时，只有即时奖励重要；智能体会短视行动。当 γ 接近 1 时，未来奖励几乎与即时奖励同样有价值；智能体会更具远见。折扣因子还决定总奖励和是否收敛；更高的 γ 会使价值函数对长期后果更敏感，也可能使学习更困难。

基于价值的方法与基于策略的方法

Q-learning 是典型的基于价值的方法：维护对最优 Q 函数的估计 Q(s, a)，在观测到转移时将其朝 bootstrapped target 更新，并根据当前估计贪婪行动（同时使用 ε-greedy 探索）。DQN 使用深度神经网络来表示 Q，并引入了两个具体的稳定化机制：经验回放缓冲区会存储并随机采样过去的转移，打破连续经验之间的相关性；目标网络会在许多步中保持 bootstrap 目标固定，防止追逐不断移动目标所带来的不稳定。

策略梯度方法直接参数化策略，并通过梯度上升优化期望累积奖励。策略梯度定理（Sutton 等，2000）给出了期望奖励相对于策略参数的梯度：∇J(θ) = E_{τ~π_θ}[Σ_t ∇log π_θ(a_t

s_t) A_t]，其中 A_t 是 advantage——也就是所采取行动的 Q 值与该状态基线价值之间的差值。Actor-critic 方法同时学习一个策略（actor）和一个价值函数（critic），用 critic 估计 advantage 并降低方差。

on-policy/off-policy 的区分是基础性的。On-policy 方法（SARSA、PPO）从当前策略生成的数据中学习当前策略。Off-policy 方法（Q-learning、SAC）可以用一个不同策略生成的数据来学习另一个策略。Off-policy 学习允许数据复用——把经验存储在回放缓冲区中，并多次采样使用——这对样本效率至关重要。带函数近似的 off-policy 学习在理论稳定性上更加脆弱：函数近似、off-policy 学习和 bootstrapping 这“三重致命组合”（deadly triad）可能导致发散，许多重要算法也因此在实践中失败。

PPO 的广泛采用，来自它在简单性、稳定性和性能之间取得的平衡。它会约束每次策略更新，避免迈出过大步伐——这个步伐由新旧策略概率之间的比率来衡量——从而防止无约束策略梯度步骤可能导致的灾难性性能退化。它的实现比 TRPO 的二阶约束更简单，同时达到相近稳定性。

信用分配、探索与样本效率

时序差分学习通过把长期问题分解成重叠的短期问题来处理信用分配。每一步都会产生一个 TD 误差——bootstrapped target 与当前估计之间的差异——这个误差会把信用沿时间向后传播。资格迹（eligibility traces）进一步推广了这一点：它按照行动发生时间的近近程度，把信用分配给最近的行动，在一步 TD（只给紧邻前一步行动分配信用）和蒙特卡洛方法（给一个回合中的所有行动均匀分配信用）之间进行插值。

n-step TD 方法的多步回报，以及 TD(λ) 的 λ 加权回报，提供了额外的信用分配机制。PPO 和现代 actor-critic 方法通常使用广义优势估计（generalized advantage estimation，GAE，Schulman 等，2016），它会计算 n-step advantages 的加权组合，以在梯度估计的偏差和方差之间取得平衡。

深度 RL 中的探索比表格设定更困难。ε-greedy 策略适用于小状态空间，因为状态会被频繁重新访问；但在大状态空间中，随机探索会把太多步浪费在无效行动上。内在动机方法会给访问新颖状态的智能体额外奖励，这种新颖性可以用学到模型的预测误差衡量（curiosity-driven exploration，Pathak 等，2017），也可以用模型集成之间的分歧衡量，或用固定随机网络的预测误差衡量（Random Network Distillation，Burda 等，2018）。这些方法使智能体能够在 Montezuma’s Revenge 这类困难探索环境中探索成功，而 ε-greedy 在这些环境中会完全失败。

学习这一部分会改变什么

强化学习会改变实践者思考序列决策，以及思考规约与行为之间关系的方式。

第一个变化，是把 MDP 框架作为建模语言来熟练使用。把一个真实问题翻译成 MDP——定义什么是状态、动作空间是什么、奖励函数应该是什么、折扣因子表示什么——本身就是一项重要分析活动。真正内化 MDP 框架的实践者，会系统性地提出这些问题，而不是把问题结构视为某个现成框架自然给定的东西。

第二个变化，是倾向于把决策看成序列性的，并思考其长期后果。监督学习问的是：“这个输入对应的正确输出是什么？”RL 问的是：“在时间中什么策略是最优的？”这种重新框定揭示了短期目标与长期目标如何冲突，为什么贪婪策略在序列场景中常常失败，以及为什么信用分配问题真的困难。真正内化这一点的实践者，在任何行动会产生下游后果的领域中，都会以不同方式思考优化目标。

第三个变化，是在经验 RL 中形成方法论纪律。Henderson 等人的 “Deep Reinforcement Learning that Matters”（2018）表明，深度 RL 论文中的结果经常不可复现：不同随机种子、不同超参数选择和不同实现细节，都可能产生质上不同的结果。学过 RL 的实践者知道，实验需要使用多个随机种子，报告方差，记录实现细节，并以适当怀疑态度对待已发表结果。这种纪律在 RL 中比在大多数机器学习领域中更重要，因为 RL 的方差在结构上更高。

第四个变化，是理解奖励规格问题是一项根本挑战。奖励函数规定智能体被要求最大化什么——而智能体会精确最大化它被要求最大化的东西，而不是人本来想要的东西。奖励黑客文献记录了许多案例：智能体会找到意料之外的方式来达成字面目标，却违背奖励的精神。这不是一个简单修复的 bug；它是目标设定错误下优化的结构性属性。理解这一点，会塑造实践者处理 RLHF、机器人目标设定，以及任何给智能体一个目标去最大化的场景的方式。

资源

书籍与文本

Sutton 和 Barto 的 Reinforcement Learning: An Introduction（第 2 版，2018，可在 incompleteideas.net 免费获取）是经典教材。它由这个领域的奠基者写成，呈现了 RL 设计者眼中的 RL——数学发展细致，直觉与形式并重，并带有历史视角，能澄清每个思想从何而来。第二版在第一版基础上加入了与深度 RL 的连接。对于任何想理解 RL 而不只是使用 RL 的人来说，完整读完 Sutton 和 Barto，并完成练习，是正确路径。其免费可获取状态也消除了任何门槛。

对于数学基础以及与最优控制的连接，Bertsekas 的 Dynamic Programming and Optimal Control（第一、二卷，Athena Scientific）和 Reinforcement Learning and Optimal Control（2019）提供了比 Sutton 和 Barto 更高层次的严谨处理。Bertsekas 的视角是：RL 和最优控制是同一主题，只是来自不同共同体；他的书会明确展示这些连接。

Agarwal、Jiang、Kakade 和 Sun 的 Reinforcement Learning: Theory and Algorithms（可在 rltheorybook.github.io 免费获取）提供了研究生水平的当代理论 RL——PAC 界、样本复杂度、函数近似稳定性理论——适合想要理论严谨性的学习者。

Lattimore 和 Szepesvári 的 Bandit Algorithms（Cambridge，2020，可在 tor-lattimore.com 免费获取）以完整数学严谨性覆盖 bandit 问题——也就是没有状态的决策这一 RL 特例。Bandit 既是理解探索理论的前置基础，也是一个重要应用领域，例如临床试验、推荐系统和 A/B 测试。

书籍	作用	类型
Sutton & Barto, Reinforcement Learning: An Introduction（第 2 版，免费）	经典基础教材	入门
Bertsekas, Reinforcement Learning and Optimal Control	最优控制连接；数学深度	深入
Bertsekas, Dynamic Programming and Optimal Control（两卷）	动态规划基础的深入处理	参考
Agarwal, Jiang, Kakade & Sun, RL: Theory and Algorithms（免费）	当代理论	深入
Lattimore & Szepesvári, Bandit Algorithms（免费）	Bandit 理论	深入
Puterman, Markov Decision Processes	MDP 数学基础	参考

课程与讲座

Berkeley CS285（Deep Reinforcement Learning，Sergey Levine，YouTube 免费讲座）是深度 RL 的标准研究生课程。讲座以当代文献所需的深度覆盖算法和理论基础。作业涉及在标准基准上实现标准深度 RL 算法，这是合适的实践入门。

David Silver 的 RL 讲座系列（DeepMind/UCL，YouTube 免费）以极高的清晰度覆盖 RL 基础材料——MDP、动态规划、TD 学习、函数近似——至今仍是理论基础最好的视频入门。

Sutton 在 University of Alberta 的在线课程（Coursera，可免费旁听）由教材作者提供结构化教学，并以教材作为伴读材料。

课程	平台	类型
Berkeley CS285 Deep RL（免费）	YouTube / course site	入门
David Silver RL lectures（免费）	YouTube	入门
Sutton’s Alberta RL course（可免费旁听）	Coursera	入门
Hugging Face Deep RL Course（免费）	huggingface.co/learn	实践

实践、工具与当前资料

OpenAI Spinning Up（spinningup.openai.com 免费）既提供深度 RL 入门，也提供核心算法（VPG、TRPO、PPO、DDPG、TD3、SAC）在 PyTorch 中的清晰实现。这些实现可读性强，可以作为正确实现这些算法的经典参考。完整学习 Spinning Up，并至少对照参考实现从零实现一个算法，是合适的实践入口。

Gymnasium（OpenAI Gym 的维护分支，gymnasium.farama.org 免费）提供标准 RL 环境 API，包含经典控制、Atari、MuJoCo 和其他环境。几乎所有深度 RL 算法都会在 Gymnasium 兼容环境上评估；熟悉这个 API 在实践上很有必要。

Stable Baselines3（免费）提供 DQN、PPO、SAC、TD3、A2C 和其他标准算法的生产级实现，并具有一致 API。它适合用于基线比较，也适合在不重新实现算法的情况下，把标准算法应用到新环境。

Henderson 等人的 “Deep Reinforcement Learning that Matters”（NeurIPS 2018，免费）是任何严肃经验 RL 工作之前的必读文献。它记录了已发表深度 RL 结果中的可复现性问题，并规定了可靠经验 RL 所需的方法论纪律——多个随机种子、完整超参数报告、统计显著性。

资源	平台	类型
OpenAI Spinning Up（免费）	spinningup.openai.com	实践
Gymnasium（免费）	gymnasium.farama.org	实践
Stable Baselines3（免费）	stable-baselines3.readthedocs.io	实践
CleanRL（免费）	GitHub	实践
RL Baselines3 Zoo（免费）	rl-baselines3-zoo.readthedocs.io	实践
Safety Gym（免费）	openai.com	辅助
Henderson et al., “Deep RL that Matters”（免费）	深度 RL 可复现性里程碑	深入

陷阱

陷阱	为什么会误导	更好的回应
跳过表格型 RL，直接进入深度 RL	DQN 论文和惊人的 Atari 结果，让许多学习者把深度 RL 当作入口，从而绕过提供概念基础的表格型 RL。结果是，实践者可以实例化 DQN，却解释不了为什么需要回放缓冲区，无法识别 Q-learning 什么时候会发散，也无法在深度 RL 失败时调整方法。	在进入深度 RL 之前，先学习 Sutton 和 Barto 的表格型章节。在 GridWorld 这类环境中从零实现 Q-learning 和 SARSA。先理解表格设定中的理论收敛结果，再把深度 RL 近似的经验成功当作解释。
把奖励设计当成小的实现细节	指定奖励函数，就是指定目标；智能体会精确优化它被给定的东西。奖励黑客——即智能体通过违背奖励意图的行为获得高奖励——已经在文献和生产级 RL 部署中被记录。奖励设计错误，会得到一个满足字面要求、却违背精神的智能体。	在设计奖励函数前，学习奖励黑客文献。像对待 API 契约一样认真对待奖励规格：这个奖励会激励哪些非预期行为？一个足够强的智能体可能找到哪些方式来利用这个奖励？阅读 Krakovna 等人的 specification gaming examples repository。
忽略可复现性问题	已发表的深度 RL 结果经常无法在不同随机种子、实现和超参数配置下稳定复现。不考虑这一点的实践者，会把随机波动误认为算法之间的有意义差异。	实验至少使用 5–10 个随机种子，报告均值和标准差或置信区间，不要从单次运行中得出结论。阅读 Henderson 等人（2018），并把其中的方法论建议当作最低标准，而不是可选项。
在监督学习更合适的地方使用 RL	当问题确实具有序列结构——决策会影响未来状态——并且可以进行环境交互时，RL 才合适。许多看起来像 RL 的问题（推荐、带不确定反馈的分类）更适合用历史数据上的监督学习、bandit 算法或离线 RL 解决。在交互昂贵或危险的地方使用在线 RL，会不必要地增加样本低效和安全风险。	先问这个问题是否真的需要在线交互和序列决策。如果问题可以被框定为历史数据上的监督学习，就先这样做。如果奖励信号立即可用，并且行动不影响未来状态，就使用 bandit 方法，而不是完整 RL。
把 RLHF 当作已经解决的对齐	与单纯预训练相比，RLHF 显著改善了语言模型行为；它的成功使一些实践者把对齐看成一个已有工程解法的问题。实际上，RLHF 产生的是对训练数据中表达的特定偏好的对齐——这些偏好可能反映标注者偏见，可能奖励迎合性回答，也可能无法泛化到新情境或更高能力水平。	把 RLHF 看作一种有已知限制的工具，而不是解决方案。学习已记录的失败模式：迎合性（模型同意用户而不是追求准确）、模式崩塌（回答多样性降低）和过度优化（Goodhart 定律作用于奖励模型）。阅读 DPO 论文，理解另一种方法。也要明白，对齐研究仍在进行，当前方法只是部分解决方案。
忽视样本效率要求	在线深度 RL 往往需要数百万次环境交互才能学会——比人类完成类似任务所需的经验多出数个数量级。在物理机器人、医疗治疗、金融决策这类交互昂贵的场景中，如果不考虑样本效率就应用在线 RL，会得到理论上可行、实践上不可行的系统。	在把 RL 应用于新领域之前，评估环境交互的成本和可用样本预算。如果在线交互有限，可以考虑从日志数据中进行离线 RL、从示范中进行模仿学习、用基于模型的 RL 减少所需样本，或使用混合方法。样本效率是设计约束，不是实现细节。