English
Reinforcement learning studies how to learn decisions from experience when actions affect later observations and rewards. The learner may collect that experience through interaction or use an existing dataset. It seeks a policy: a rule for choosing actions from the information available to the agent.
The distinctive difficulty is sequential feedback. An action can change both the eventual outcome and what the learner gets to observe next. Reward may arrive long after the relevant choice, so the agent must estimate long-term consequences, explore uncertain alternatives, and learn from incomplete feedback. Predicting a label and choosing an action are different tasks even when both use neural networks.
RL connects dynamic programming, optimal control, probability, and learning. A known environment model permits planning; an unknown model requires learning from data, possibly learning a model first and then planning with it. This distinction links the search methods in §5.1 to value functions and policy learning here.
Prerequisites: Expectations and conditional probabilities (§2.5), basic optimization (§5.2), and dynamic programming (§2.6). Deep learning (§5.3) is needed for deep RL, but small tabular problems provide the clearest starting point.
From Bellman’s Equations to Language Model Alignment
RL grew from several traditions. Psychological studies examined learning through consequences; optimal control and Bellman’s dynamic programming formalized sequential decision problems; early AI programs experimented with learning from play. Nilsson traces these overlapping contributions. There was no single clean progression from a completed theory to its later applications.
Bellman’s value equations decompose a long-term objective into immediate reward and the value of a successor state. If transition probabilities and rewards are known, dynamic programming can use that model directly. The difficulty is often the number of states required to represent the problem, especially when several state variables vary independently.
Samuel’s checkers program learned aspects of position evaluation while using search and domain-specific design. Sutton’s temporal-difference work developed learning from the discrepancy between successive predictions: a prediction can be updated toward an observed reward plus a later estimate before the final outcome arrives. This bootstrapping offers a different bias–variance and data-use tradeoff from waiting for a completed return; it is not universally more accurate or efficient.
Watkins’s Q-learning connected sampled transitions to learning optimal action values. Its tabular convergence results require conditions such as adequate visits to state–action pairs and suitable learning rates, along with the assumptions of the MDP setting. They do not transfer automatically to a deep network trained on a finite replay buffer. REINFORCE and actor–critic methods developed another route by directly adjusting action probabilities.
Research on dopamine responses found relationships with reward-prediction errors, including shifts in response as reward-predicting cues are learned. This is evidence connecting some biological learning signals to computational models, not a claim that every dopamine signal implements one RL algorithm or that an RL objective explains all behavior.
DQN combined learned visual representations with Q-learning, experience replay, and a target network. The 2015 Nature paper evaluated 49 Atari games using a common algorithm and architecture, with a separately trained agent for each game. Performance varied across games; one jointly trained model did not outperform humans on all 49. The significant result was that one learning setup could address many tasks from high-dimensional observations.
AlphaGo combined learned policies and values with search, including human-game training and self-play. AlphaZero removed the requirement for human-game examples in its training setup while retaining game rules and substantial computation. MuZero learned a model useful for predicting reward, value, and policy during planning; it did not need to reconstruct every detail of the environment. These examples connect learning, representation, and planning rather than eliminating the need for design choices.
TRPO and PPO addressed policy-update stability. PPO’s clipping changes the optimization incentive for moving action probabilities far from the behavior policy; it is not a hard guarantee that performance will improve. Christiano and colleagues’ 2017 preference-learning work, followed by language-model applications such as InstructGPT, used learned feedback to supply an objective for RL. The preference model and the resulting policy remain separate objects to evaluate.
Offline RL studies fixed datasets, model-based RL uses learned dynamics or prediction models, and multi-agent RL studies interacting learners. Each changes the information available for improving a policy. A dataset may contain little evidence about an attractive new action; a learned model may be wrong where a planner chooses to visit; other learning agents may change the effective environment. These are central learning problems, not details a larger network automatically resolves.
The MDP Framework, Value Functions, and Policy Learning
Markov Decision Processes: The Formal Foundation
A finite discounted MDP specifies states, available actions, transition probabilities P(s′|s,a), expected immediate reward R(s,a), and a discount 0 ≤ γ < 1. A policy π(a|s) gives action probabilities. The Markov assumption says the state contains the history needed to predict the next transition and reward given an action. Whether a sensor observation contains that information is a modeling question.
The return is a sum of rewards over time, such as Gₜ = Rₜ₊₁ + γRₜ₊₂ + γ²Rₜ₊₃ + …. The state value V^π(s) is expected return when starting at s and following π; Q^π(s,a) first takes action a, then follows π. The star in V* or Q* instead denotes optimal value. Reward, sampled return, estimated value, and optimal value should not be used interchangeably.
The optimality equation is V*(s) = max_a [R(s,a) + γ Σ_s′ P(s′|s,a)V*(s′)]. In a finite MDP with bounded rewards and discount below one, the Bellman optimality operator is a contraction and has a unique value-function fixed point. Value iteration approaches it; an exact optimum need not be reached after a fixed finite number of iterations. Episodic undiscounted and average-reward objectives use different conditions.
For a small example, one action ends the episode with reward 3. Another gives reward 0 now and leads to a state where the next action ends with reward 5. With γ=0.9, the second choice is worth 0 + 0.9 × 5 = 4.5, so it is preferable despite giving no immediate reward. This example can be solved by planning if the transitions are known; learning is needed when their values must be estimated from experience.
A partially observed process can still have Markovian underlying states. The agent may need a history summary or a belief distribution over states to choose well. If two situations look identical but require different actions, simply using the current observation as the MDP state can be inadequate. Adding memory changes the representation available to the policy.
Value-Based and Policy-Based Methods
A tabular Q-learning update moves an estimate toward y = r + γ max_a′ Q(s′,a′): Q(s,a) ← Q(s,a) + α[y−Q(s,a)]. At a terminal transition the target is just the reward. If the old estimate is 2, reward is 1, the next state’s largest estimate is 4, γ=0.9, and α=0.5, the target is 4.6 and the new estimate is 3.3. The target includes an estimate, so errors can propagate as well as useful information.
DQN represents Q-values with a neural network. Experience replay reuses stored transitions and reduces dependence between nearby training samples; it does not make the entire dataset independent. A slowly updated target network reduces how rapidly bootstrap targets change. These mechanisms help training, without extending tabular convergence guarantees to every network or dataset.
Policy-gradient methods instead adjust a parameterized policy directly. For a finite-horizon undiscounted episodic objective, one form of the gradient is E[Σₜ ∇θ log πθ(aₜ|sₜ)(Gₜ−b(sₜ))], where Gₜ is the remaining sampled return. An action-independent baseline b(sₜ) can reduce variance without changing the expected gradient under the usual assumptions. Actor–critic methods learn a value estimate for this purpose and often use bootstrapped advantage estimates, introducing their own bias–variance tradeoff. Discounted objectives require matching discount conventions in the estimator.
On-policy learning evaluates or improves the policy supplying the relevant data. Off-policy learning can use data from another behavior policy, with algorithm-specific requirements. It enables reuse but does not make arbitrary old data sufficient: coverage and distribution mismatch matter. Function approximation, bootstrapping, and off-policy training can interact to cause instability or divergence.
PPO commonly collects data with a policy, then performs a limited number of updates using ratios of new to old action probabilities. Its clipped objective discourages some excessively large changes. Clipping is not a hard constraint on every ratio or on total policy divergence, and does not guarantee monotonic improvement. The PPO documentation explains why implementations also monitor approximate KL divergence and may stop an update early.
Credit Assignment, Exploration, and Sample Efficiency
One-step TD updates the value of a visited state using the next reward and value estimate. Repeated experience can carry information back to earlier states. Multi-step returns include more observed rewards before bootstrapping. Eligibility traces keep a decaying record of past visits, allowing a TD error to update multiple earlier estimates; TD(λ) relates this to a weighted mixture of return lengths. These methods differ in variance, bias, and when information becomes available.
Exploration determines which evidence is collected. An ε-greedy policy occasionally chooses a random action, but sparse-reward problems may require a coordinated sequence that such sampling rarely discovers. Optimism, uncertainty estimates, count-based bonuses, or learned novelty signals can encourage exploration. A prediction-error bonus can also reward inherently unpredictable observations, so “surprising” is not automatically “useful to learn.”
Sample efficiency measures learning relative to environment experience, which differs from computation time. A model-based method may use extra computation to learn and plan through a dynamics model, saving interactions when that model is useful. Planning can also exploit model errors. Offline RL faces a related problem when a policy chooses actions poorly represented in its fixed dataset: high predicted value may reflect extrapolation rather than evidence.
Evaluate the interaction budget, computation, reward definition, and policies used to collect data. Keep evaluation episodes separate from training exploration and report results across seeds and tasks. Distinguish actual episode termination from a time-limit truncation when constructing bootstrap targets. That implementation detail changes the learning problem.
What Studying This Changes
You should be able to define state, observation, action, reward, policy, and return for a small task, and explain what is known versus learned. A simulator does not remove modeling assumptions; it makes them executable.
Solve a tiny MDP by value iteration, then learn it with tabular Q-learning and compare the estimates with the known solution. Change the discount, reward timing, exploration, and observation representation separately. This shows which changes alter the objective and which change the learner’s ability to find a solution.
Treat reward design and evaluation as separate responsibilities. A learner may exploit a proxy, fail to optimize it, or encounter a different environment. “The agent maximizes exactly what it was told” overlooks optimization and generalization failures. Inspect trajectories and test whether improved return corresponds to the intended result, especially when using learned feedback.
Resources
Books and Texts
Sutton and Barto’s Reinforcement Learning: An Introduction develops bandits, dynamic programming, Monte Carlo methods, temporal-difference learning, and function approximation. Start with the tabular examples, where you can inspect every value and compare an update with its equation. Bertsekas’s books connect these ideas to optimal control and give a more mathematical treatment.
Agarwal, Jiang, Kakade, and Sun’s Reinforcement Learning: Theory and Algorithms develops sample-complexity and learning guarantees under explicit assumptions. Lattimore and Szepesvári’s Bandit Algorithms isolates exploration in problems without action-dependent state transitions. This simpler setting helps separate exploration from the additional difficulty of long-term control.
| Book | Role | Type |
|---|---|---|
| Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., free) | Canonical foundational text | Entry |
| Bertsekas, Reinforcement Learning and Optimal Control | Optimal control connection; mathematical depth | Depth |
| Bertsekas, Dynamic Programming and Optimal Control (2 vols.) | DP foundations at depth | Reference |
| Agarwal, Jiang, Kakade & Sun, RL: Theory and Algorithms (free) | Contemporary theory | Depth |
| Lattimore & Szepesvári, Bandit Algorithms (free) | Bandit theory | Depth |
| Puterman, Markov Decision Processes | MDP mathematical foundation | Reference |
Courses and Lectures
David Silver’s RL lectures cover the foundational sequence from MDPs to value and policy methods. The University of Alberta’s Reinforcement Learning Specialization offers a course sequence connected to Sutton and Barto’s textbook; check the provider’s current access conditions. Berkeley CS285 goes further into deep RL and assumes a stronger ML background. Its implementation assignments are useful after the tabular methods are clear.
| Course | Platform | Type |
|---|---|---|
| Berkeley CS285 Deep RL (free) | YouTube / course site | Entry |
| David Silver RL lectures (free) | YouTube | Entry |
| University of Alberta Reinforcement Learning Specialization | Coursera; check access conditions | Entry |
| Hugging Face Deep RL Course (free) | huggingface.co/learn | Practice |
Practice, Tools, and Current Sources
Spinning Up explains core deep RL algorithms and provides reference implementations. Gymnasium supplies an environment interface, while Stable Baselines3 and CleanRL offer implementations for comparison. Before interpreting a score difference, check observation preprocessing, reward transformations, episode limits, and the distinction between termination and truncation.
Henderson et al.’s “Deep Reinforcement Learning that Matters” discusses how implementation and experimental choices affect reported results. Use multiple seeds and report the evaluation protocol and variation across runs. For constrained-control experiments, Safety-Gymnasium provides tasks with costs as well as rewards; success on these simulated constraints does not establish safety in a physical deployment.
| Resource | Platform | Type |
|---|---|---|
| OpenAI Spinning Up (free) | spinningup.openai.com | Practice |
| Gymnasium (free) | gymnasium.farama.org | Practice |
| Stable Baselines3 (free) | stable-baselines3.readthedocs.io | Practice |
| CleanRL (free) | GitHub | Practice |
| RL Baselines3 Zoo (free) | rl-baselines3-zoo.readthedocs.io | Practice |
| Safety-Gymnasium (free) | safety-gymnasium.readthedocs.io | Auxiliary |
| Henderson et al., “Deep RL that Matters” (free) | Deep RL reproducibility landmark | Depth |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Starting with deep RL before a tabular example | Approximation and optimization can obscure an incorrect return or transition model. | Solve a small MDP exactly and compare the learned values. |
| Treating reward design as a minor detail | The reward and horizon determine which tradeoffs the policy is trained to make. | Inspect trajectories and evaluate the intended outcome separately from return. |
| Reporting only a favorable run | Seeds, implementation, and evaluation conditions can change the result. | Document settings and report variability across runs and tasks. |
| Using RL whenever an output must be chosen | Some tasks need prediction or planning with a known model rather than interaction-based learning. | Identify delayed consequences, available feedback, and simpler alternatives. |
| Treating preference reward as solved alignment | The reward model can be biased or inaccurate where the policy seeks high scores. | Evaluate policy outcomes independently and inspect reward-model failures. |
| Ignoring experience costs and coverage | A large number of updates cannot create evidence about unseen actions. | Report interaction budgets and test policies outside well-covered training cases. |
中文
强化学习研究怎样从经验中学习决策，其中动作会影响后续观测与奖励。学习者可以通过交互收集经验，也可以使用已有数据集。它要学习策略：根据智能体可获得的信息选择动作的规则。
其特有困难是序贯反馈。动作既可能改变最终结果，也可能改变学习者接下来能观测到什么。奖励可能在相关选择之后很久才出现，因此智能体需要估计长期后果、探索不确定选项，并从不完整反馈中学习。即使都使用神经网络，预测标签与选择动作仍是不同任务。
强化学习连接动态规划、最优控制、概率与学习。已知环境模型时可以规划；模型未知时需要从数据中学习，也可以先学习模型，再利用它规划。这个区别把 §5.1 的搜索方法与本节的价值函数、策略学习联系起来。
前置知识：期望与条件概率（§2.5）、基本优化（§5.2）、动态规划（§2.6）。深度强化学习需要 §5.3，但小型表格问题是更清楚的起点。
从 Bellman 方程到语言模型对齐
强化学习来自多种传统。心理学研究行为后果如何影响学习，最优控制与 Bellman 的动态规划形式化序贯决策问题，早期 AI 程序则探索从对弈中学习。Nilsson 追溯了这些相互重叠的贡献，其发展并不是从完成的理论单线走向后来的应用。
Bellman 价值方程把长期目标分解为即时奖励与后继状态的价值。如果转移概率和奖励已知，动态规划可以直接使用模型。困难往往是表示问题所需的状态数，尤其当多个状态变量可以独立变化时。
Samuel 的跳棋程序在使用搜索与领域设计的同时，学习部分局面评价。Sutton 的时间差分研究发展了从连续预测之间的差异学习的方法：在最终结果尚未到来时，就可以用观测奖励加上后续估计来更新先前预测。这种自举带来与等待完整回报不同的偏差—方差及数据使用取舍，并不普遍更准确或更高效。
Watkins 的 Q-learning 把采样转移与最优动作价值学习联系起来。表格形式的收敛结果除 MDP 设置的假设外，还要求状态—动作对被充分访问，以及合适学习率等条件。它们不会自动适用于有限回放缓冲区上训练的深度网络。REINFORCE 与 actor–critic 方法则发展了直接调整动作概率的另一条路线。
多巴胺反应研究发现了与奖励预测误差的联系，包括学会预测奖励的线索后，反应发生相应转移。这些证据把部分生物学习信号与计算模型连接起来，并不意味着每个多巴胺信号都实现同一个 RL 算法，也不意味着一个 RL 目标能解释全部行为。
DQN 把学习得到的视觉表示与 Q-learning、经验回放、目标网络结合起来。2015 年 Nature 论文在 49 款 Atari 游戏上使用共同的算法与架构，但为每款游戏分别训练智能体。各游戏表现不同，并不是一个联合训练的模型在全部 49 款游戏上都超过人类。重要结果是同一种学习设置能够从高维观测出发处理多种任务。
AlphaGo 结合学出的策略、价值与搜索，训练包含人类棋谱和自我对弈。AlphaZero 在其训练设置中取消了对人类棋谱例子的要求，仍然使用游戏规则与大量计算。MuZero 学习适合规划的模型，用来预测奖励、价值和策略，不必重建环境的每个细节。这些例子把学习、表示与规划连接起来，并没有消除设计选择。
TRPO 与 PPO 处理策略更新的稳定性。PPO 裁剪改变了使动作概率远离行为策略的优化激励，并不硬性保证性能改善。Christiano 等人 2017 年的偏好学习工作，以及后来的 InstructGPT 等语言模型应用，用学出的反馈为强化学习提供目标。偏好模型与由此得到的策略仍是需要分别评估的对象。
离线强化学习研究固定数据集，基于模型的强化学习使用学出的动力学或预测模型，多智能体强化学习研究相互作用的学习者。它们分别改变了改善策略所能使用的信息：数据集可能对某个看似有利的新动作提供很少证据，学习模型可能在规划器选择访问的位置出错，其他学习者可能改变实际环境。这些是核心学习问题，不会被更大网络自动解决。
MDP 框架、价值函数与策略学习
马尔可夫决策过程：形式基础
有限折扣 MDP 规定状态、可用动作、转移概率 P(s′|s,a)、期望即时奖励 R(s,a)，以及折扣 0 ≤ γ < 1。策略 π(a|s) 给出动作概率。Markov 假设要求：给定动作后，状态包含预测下一次转移与奖励所需的历史信息。传感器观测是否包含这些信息，是建模问题。
回报是跨时间的奖励之和，例如 Gₜ = Rₜ₊₁ + γRₜ₊₂ + γ²Rₜ₊₃ + …。状态价值 V^π(s) 是从 s 出发并遵循 π 的期望回报；Q^π(s,a) 则先执行动作 a，之后遵循 π。V* 或 Q* 中的星号表示最优价值。奖励、采样回报、价值估计与最优价值不能混用。
最优性方程为 V*(s) = max_a [R(s,a) + γ Σ_s′ P(s′|s,a)V*(s′)]。在奖励有界、折扣小于一的有限 MDP 中，Bellman 最优性算子是压缩映射，具有唯一的价值函数不动点。价值迭代逐渐逼近它，不必在固定的有限次迭代后精确达到最优。无折扣回合任务与平均奖励目标使用不同条件。
一个小例子中，动作一以奖励 3 结束回合；动作二当前奖励为 0，进入下一状态后再执行一个动作，就以奖励 5 结束。取 γ=0.9，第二种选择价值为 0 + 0.9 × 5 = 4.5，因此虽无即时奖励，却更值得选择。如果转移已知，可以通过规划解出这个例子；需要根据经验估计其价值时，才出现学习问题。
部分可观测过程的底层状态仍然可以具有 Markov 性。智能体可能需要历史摘要或对状态的信念分布，才能作出合适选择。如果两个情形看起来相同，却要求不同动作，仅把当前观测当作 MDP 状态可能不够。增加记忆会改变策略可以使用的表示。
基于价值的方法与基于策略的方法
表格 Q-learning 把估计移向 y = r + γ max_a′ Q(s′,a′)：Q(s,a) ← Q(s,a) + α[y−Q(s,a)]。终止转移的目标只有奖励。如果旧估计为 2，奖励为 1，下一状态最大估计为 4，γ=0.9、α=0.5，那么目标为 4.6，新估计为 3.3。目标本身包含估计，因此既会传播有用信息，也会传播误差。
DQN 用神经网络表示 Q 值。经验回放复用存储的转移，降低相邻训练样本之间的依赖，并不使整个数据集完全独立。缓慢更新的目标网络减少自举目标变化的速度。这些机制能帮助训练，但不会把表格形式的收敛保证扩展到所有网络或数据集。
策略梯度方法直接调整参数化策略。对于有限时域、无折扣的回合目标，一种梯度形式是 E[Σₜ ∇θ log πθ(aₜ|sₜ)(Gₜ−b(sₜ))]，其中 Gₜ 是剩余的采样回报。在通常假设下，不依赖动作的基线 b(sₜ) 可以降低方差而不改变梯度期望。Actor–critic 方法为此学习价值估计，并常使用自举优势估计，产生相应的偏差—方差取舍。折扣目标则需要估计器采用相应的折扣约定。
同策略学习评估或改善提供相关数据的策略；异策略学习可以使用其他行为策略产生的数据，但有各算法自己的要求。它允许复用数据，却不使任意旧数据都足够：覆盖范围与分布不匹配仍然重要。函数逼近、自举和异策略训练可能相互作用，导致不稳定或发散。
PPO 通常先用一个策略收集数据，再根据新旧动作概率之比进行有限次更新。裁剪目标抑制部分过大的变化，但并不硬性限制每个概率比或整个策略的散度，也不保证单调改善。PPO 文档说明了为何实现还会监控近似 KL 散度，并可能提前停止一轮更新。
信用分配、探索与样本效率
单步 TD 用下一次奖励和价值估计更新已访问状态的价值。反复经验可以把信息传回更早状态；多步回报在自举之前纳入更多观测奖励。资格迹保留逐渐衰减的访问记录，使一个 TD 误差能够更新多个先前估计；TD(λ) 把它与不同回报长度的加权组合联系起来。这些方法在方差、偏差及信息何时可用方面不同。
探索决定收集什么证据。ε-greedy 策略偶尔随机选择动作，但稀疏奖励问题可能要求一串协调动作，随机采样很少发现它。乐观估计、不确定性估计、访问计数奖励或学出的新颖性信号可以鼓励探索。预测误差奖励也可能奖励本来就不可预测的观测，因此“令人意外”不自动等于“值得学习”。
样本效率按环境经验衡量学习，不同于计算时间。基于模型的方法可以花额外计算学习动力学模型并进行规划，在模型有用时节省交互；规划也可能利用模型错误。离线强化学习面临类似问题：当策略选择固定数据集中缺少覆盖的动作时，高预测价值可能反映外推，而非证据。
评估时应说明交互预算、计算量、奖励定义，以及收集数据所用策略。把评估回合与训练探索分开，报告多个随机种子和任务的结果。构造自举目标时，还要区分真正的回合终止与时间上限截断；这个实现细节会改变学习问题。
学习这一部分会改变什么
学完后，应能为小任务定义状态、观测、动作、奖励、策略和回报，并说明哪些内容已知、哪些需要学习。模拟器没有消除建模假设，而是使假设可以执行。
先用价值迭代解一个小 MDP，再用表格 Q-learning 学习它，把估计与已知解比较。分别改变折扣、奖励时机、探索与观测表示，就能看见哪些变化改变目标，哪些变化改变学习者找到解的能力。
把奖励设计与评估视为不同责任。学习者可能利用代理指标，可能没有优化好它，也可能遇到不同环境。“智能体精确最大化了被告知的目标”忽略了优化与泛化失败。应检查轨迹，检验回报提高是否对应预期结果，尤其在反馈本身也是学出来的时候。
资源
书籍与文本
Sutton 与 Barto 的 Reinforcement Learning: An Introduction 依次讲多臂老虎机、动态规划、蒙特卡洛方法、时序差分学习和函数近似。从表格型例子开始，可以检查每一个值，并将一次更新与公式逐项对应。Bertsekas 的书将这些思想与最优控制联系起来，并给出更深入的数学处理。
Agarwal、Jiang、Kakade 与 Sun 的 Reinforcement Learning: Theory and Algorithms 在明确假设下讨论样本复杂度与学习保证。Lattimore 与 Szepesvári 的 Bandit Algorithms 在没有由动作决定的状态转移的问题中研究探索。这种较简单的设置，有助于将探索与长期控制增加的困难分开理解。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Sutton & Barto, Reinforcement Learning: An Introduction（第 2 版，免费） | 经典基础教材 | 入门 |
| Bertsekas, Reinforcement Learning and Optimal Control | 最优控制连接；数学深度 | 深入 |
| Bertsekas, Dynamic Programming and Optimal Control（两卷） | 动态规划基础的深入处理 | 参考 |
| Agarwal, Jiang, Kakade & Sun, RL: Theory and Algorithms（免费） | 当代理论 | 深入 |
| Lattimore & Szepesvári, Bandit Algorithms（免费） | Bandit 理论 | 深入 |
| Puterman, Markov Decision Processes | MDP 数学基础 | 参考 |
课程与讲座
David Silver 的 RL 讲座 从 MDP 讲到价值方法和策略方法。阿尔伯塔大学的强化学习专项课程 提供与 Sutton、Barto 教材相关的课程顺序；访问条件以课程平台的现行说明为准。Berkeley CS285 进一步讲深度 RL，需要较扎实的机器学习基础。理解表格型方法后，可以通过其中的实现作业继续学习。
| 课程 | 平台 | 类型 |
|---|---|---|
| Berkeley CS285 Deep RL（免费） | YouTube / course site | 入门 |
| David Silver RL lectures（免费） | YouTube | 入门 |
| 阿尔伯塔大学强化学习专项课程 | Coursera；访问条件见平台说明 | 入门 |
| Hugging Face Deep RL Course（免费） | huggingface.co/learn | 实践 |
实践、工具与当前资料
Spinning Up 解释主要的深度 RL 算法，并提供参考实现。Gymnasium 提供环境接口，Stable Baselines3 和 CleanRL 提供可用于比较的算法实现。解释分数差异之前，先核对观察预处理、奖励变换、回合长度限制，以及终止与截断的区别。
Henderson 等人的 “Deep Reinforcement Learning that Matters” 讨论实现和实验选择如何影响报告的结果。使用多个随机种子，并报告评估流程和各次运行之间的差异。研究约束控制时，Safety-Gymnasium 提供同时包含奖励与成本的任务；满足这些模拟约束，并不能证明物理部署是安全的。
| 资源 | 平台 | 类型 |
|---|---|---|
| OpenAI Spinning Up（免费） | spinningup.openai.com | 实践 |
| Gymnasium（免费） | gymnasium.farama.org | 实践 |
| Stable Baselines3（免费） | stable-baselines3.readthedocs.io | 实践 |
| CleanRL（免费） | GitHub | 实践 |
| RL Baselines3 Zoo（免费） | rl-baselines3-zoo.readthedocs.io | 实践 |
| Safety-Gymnasium（免费） | safety-gymnasium.readthedocs.io | 辅助 |
| Henderson et al., “Deep RL that Matters”（免费） | 深度 RL 可复现性里程碑 | 深入 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 跳过表格例子直接学深度 RL | 逼近与优化可能掩盖错误的回报或转移模型。 | 精确解一个小 MDP，再比较学出的价值。 |
| 把奖励设计当作小细节 | 奖励与时域决定策略被训练去作出的取舍。 | 检查轨迹，把预期结果与回报分开评估。 |
| 只报告表现好的一次运行 | 随机种子、实现与评估条件都会改变结果。 | 记录设置，报告多次运行与多个任务的变化。 |
| 凡是要选输出就使用 RL | 有些任务需要预测或已知模型上的规划，并不需要交互学习。 | 识别延迟后果、可用反馈及更简单的选择。 |
| 把偏好奖励当作已解决对齐 | 策略寻找高分的区域中，奖励模型可能偏置或失准。 | 独立评估策略结果，检查奖励模型失败。 |
| 忽略经验成本与覆盖范围 | 大量更新不能凭空产生未见动作的证据。 | 报告交互预算，检验策略在训练覆盖不足情形中的表现。 |