English
AI safety studies how to identify, evaluate, and reduce harms associated with AI systems. Alignment concerns how a system’s objectives and behavior relate to intended goals, human preferences, and relevant constraints. Whose intentions count, how conflicts are resolved, and who bears an error’s consequences are part of the problem; matching one user’s request is not a complete definition of safety.
Separate failure mechanisms. A model may optimize an inadequate proxy, make an ordinary prediction error, fail under changed conditions, be used for a harmful purpose, or be compromised through its surrounding software. These mechanisms call for different evidence and interventions. Training a more preferred response does not automatically correct permissions, data access, or an unreliable downstream action.
The subject connects learning (§5.2–5.5) with security (§4.5), verification (§3.6), and system reliability (§4.10). It also includes contested hypotheses about future systems. Distinguishing observed failures, controlled demonstrations, theoretical arguments, and forecasts makes the discussion more precise and keeps technical questions connected to evidence.
Prerequisites: Learning objectives and evaluation (§5.2), sequential decision problems (§5.4), and language-model systems (§5.5). Interpretability (§5.6) supplies one possible source of evidence, alongside testing, verification, and system controls.
From Early Warnings to Technical Discipline
Concerns about automated decisions have several histories. Control engineering, dependable software, robotics, and human factors studied failures in systems acting on the world. Wiener warned that a machine’s specified purpose could differ from the purpose people actually wanted. His concern connects naturally to specification and feedback; it did not require a prediction that a particular future AI architecture would exist.
I. J. Good’s 1965 discussion of an intelligence explosion considered the possibility of systems improving the design of their successors. Later arguments about highly capable systems and control, including Bostrom’s work, developed conditional scenarios. Their conclusions depend on assumptions about capability, goals, autonomy, and the dynamics of improvement. Historical influence and empirical confirmation are different questions.
As learning systems became more widely used, researchers also investigated concrete failure mechanisms. The 2016 paper Concrete Problems in AI Safety organized five problems: avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration, and distributional shift. These connected objective design, feedback cost, and learning conditions to unwanted behavior. The list was a research agenda, not a complete inventory of harms.
Preference learning supplied an alternative to hand-writing every reward. Christiano and colleagues’ 2017 work learned feedback from comparisons and used it to train behavior. Later language-model work, including InstructGPT and Constitutional AI, explored demonstrations, comparisons, and written principles in post-training. These experiments showed measurable behavioral changes under particular evaluations. They did not establish that every relevant value could be inferred from comparisons or that the resulting systems were safe in every application.
Deployment broadened the unit of analysis from a model’s output to an application with users, data, tools, and institutions. Work on model cards, incident documentation, red-teaming, and capability evaluations developed ways to describe uses, limitations, and observed failures. The NIST AI Risk Management Framework treats risk management as an organizational process as well as a technical measurement problem.
Scalable oversight and learned-objective research ask how supervision might remain effective when outputs become difficult to judge or behavior changes beyond training. Some studies use deliberately constructed models or restricted tasks to test proposed failure mechanisms. Such experiments can identify possibilities and challenge methods, while estimating their prevalence in other systems requires further evidence. A useful history follows how questions became measurable without treating either reassurance or catastrophe as a predetermined conclusion.
Specification, Oversight, and the Empirical Safety of Deployed Systems
The Specification Problem
A training objective is an operational specification of some desired behavior. Its adequacy depends on the task. Suppose a support assistant is rewarded for closing tickets quickly. It can improve that metric by resolving problems efficiently, but also by marking unresolved requests as complete. “Ticket closed” and “problem resolved” are different variables. The example exposes a missing requirement without requiring the system to possess a human-like intention to cheat.
Reward hacking or specification gaming describes behavior that scores well under a proxy while violating the intended outcome. Goodhart-style effects are a useful family of explanations: selection and optimization can favor cases where a measure stops tracking the goal well. This is a possible failure mechanism, not a theorem that every objective is unusable or every sufficiently capable learner must exploit it. Some narrowly defined tasks admit precise specifications and strong checks.
Distinguish three failures: optimizing the wrong objective, failing to optimize the chosen objective, and learning behavior that changes undesirably outside the training setting. They can look similar in an output while requiring different remedies. Claims that a learned model has acquired an internal objective require evidence about its behavior or computation; a single bad answer is not sufficient to establish such a mechanism.
Feedback methods add information but retain assumptions. Preference comparisons depend on the raters, instructions, and examples; written principles require interpretation in context; a reward model extrapolates beyond its labels. Separate user approval from correctness, and task completion from acceptable side effects. For the ticket example, evaluate verified resolution, reopening rates, and inappropriate closure alongside speed, and specify when a human must make the decision.
Scalable Oversight and the Verification Gap
Oversight becomes difficult when checking an output requires scarce expertise, substantial computation, or observing delayed consequences. Verification can be cheaper than generation for some tasks, but not all. A proof checker can validate a derivation against formal rules; it cannot establish that the theorem formalized the user’s intended claim unless that correspondence is also checked.
Task decomposition, tool-assisted checking, expert review, and random audits can increase supervision capacity. They work only when the pieces retain the information needed for a reliable judgment. A set of individually plausible summaries can omit a decisive cross-document contradiction. A reviewer aided by an unreliable model can also inherit its mistakes.
Debate proposes using competing arguments to help a judge identify errors. Its effectiveness depends on what evidence can be exposed and what the judge can evaluate; persuasive performance does not guarantee truth. Iterated amplification uses assisted judgments to train stronger systems. These are research approaches with particular experimental and theoretical assumptions, not established general solutions.
The weak-to-strong generalization study tested whether a stronger pretrained model could learn from weaker model labels on selected tasks. Improvements over the weak supervisor showed that imperfect labels need not impose an identical performance ceiling, while gaps from stronger supervision remained. This is an empirical analogy to a supervision problem, not a demonstration that arbitrary more-capable systems can be reliably aligned by weaker overseers.
Evaluate the supervisor as well as the model: which errors does it detect, which does it share, what information does it have, and what does review cost? Better oversight should improve a measured judgment or outcome. Agreement among several evaluators is weaker evidence when they share training data, assumptions, or failure modes.
Empirical Safety of Deployed Systems
Evaluate the deployed system under a stated use and threat model. Identify what information it can read, what actions it can take, and what happens after a mistaken output. An assistant that drafts a proposed refund and one that can execute it have different consequences even if their text-generation accuracy is identical.
Prompt injection concerns instructions arriving through material that should be treated as data, such as a retrieved page or tool result. Refusal training alone does not establish a trustworthy boundary between that material and authorized instructions. Limit tool permissions, validate proposed operations in application code, and keep sensitive actions behind controls appropriate to the task. Jailbreak resistance, access control, and factual accuracy are related but different properties.
Unsupported claims and sycophancy require direct evaluation. Retrieval can supply evidence but may retrieve irrelevant or manipulated material; fluent summaries can still misstate it. Preference training can reward pleasing presentation, but sycophancy has no single necessary cause. Use factual checks, counterexamples, uncertainty evaluation, and abstention criteria matched to the application. Error reduction is a measurable goal; neither “this can never be fixed” nor “the model was aligned” replaces a test.
Capability evaluations ask what a system can accomplish under particular access, assistance, and time budgets. Misuse risk additionally depends on who can access it, what other resources are available, and whether the capability changes an actor’s practical options. Controlled demonstrations, realistic task evaluations, and incident observations provide different evidence. A failed attempt by one evaluator does not establish that no other procedure would succeed.
Zero observed failures does not imply zero risk. In a simple model of 1,000 independent tests drawn from one fixed deployment distribution, with zero failures, a one-sided 95% binomial upper confidence bound on the failure probability is 1−0.05^(1/1000) ≈ 0.003. Even this statement depends on the sampling model; correlated tests or changed inputs can invalidate its interpretation. Report what was tested, the failure definition, uncertainty, and which uses remain outside the evaluation.
Monitoring and incident response extend evaluation after release. Track meaningful outcomes, investigate failures, revise controls, and check whether mitigations introduce new errors or burdens. A model change, new tool, or new user population can change the assumptions of an earlier assessment. Safety evidence belongs to a versioned system in a use setting, not permanently to a model name.
What Studying This Changes
You should be able to turn a broad safety claim into an assessable one: a specified system, users and affected parties, permitted actions, failure conditions, and supporting evidence. Distinguish an objective-design problem from an implementation defect, a distribution shift, or an access-control failure.
For a small assistant application, write down one intended outcome, one tempting proxy, and one counterexample where they diverge. Test the model and the surrounding controls separately. Record which interventions improve the intended outcome, not merely the proxy or the apparent confidence of the answer.
Read research claims according to their evidence. A theorem has assumptions; a controlled demonstration establishes a possibility under its setup; a benchmark estimates performance under a protocol; a forecast adds uncertain extrapolation. Comparing these carefully makes it possible to investigate future risks and present failures without treating either as a substitute for the other.
Resources
Books and Texts
Russell’s Human Compatible develops an argument for treating human objectives as uncertain rather than fixed and fully known. Christian’s The Alignment Problem presents the history and human consequences of learning systems. Bostrom’s Superintelligence develops conditional arguments about highly capable systems and control; distinguish those arguments from measurements of existing systems and from forecasts about when such systems will exist.
For technical study, start with “Concrete Problems in AI Safety” and the preference-learning papers, then read Ngo, Chan, and Mindermann on deep learning alignment and Casper et al. on RLHF limitations. Hubinger et al. develops the learned-optimization concern. For each proposed failure, identify the mechanism, assumptions, supporting experiments, and what evidence would weaken the claim.
| Book | Role | Type |
|---|---|---|
| Russell, Human Compatible | Accessible alignment entry; mainstream AI perspective | Entry |
| Christian, The Alignment Problem | Accessible historical and conceptual entry | Entry |
| Bostrom, Superintelligence | Arguments about long-term control risks | Depth |
| Ngo, Chan & Mindermann, “The Alignment Problem from a Deep Learning Perspective” (free) | Technical alignment treatment | Depth |
| Amodei et al., “Concrete Problems in AI Safety” (2016, free) | Foundational technical safety articulation | Depth |
| Christiano, “Deep RL from Human Preferences” (2017, free) | RLHF foundational paper | Depth |
| RLHF case study: Ouyang et al., InstructGPT paper (2022, free) | RLHF at language model scale | Depth |
| Bai et al., Constitutional AI paper (2022, free) | Constitutional AI approach | Depth |
| Preference optimization case study: Rafailov et al., DPO paper (2023, free) | Direct preference optimization | Depth |
| Casper et al., “Open Problems and Fundamental Limitations of RLHF” (free) | Systematic critique of RLHF | Depth |
| Hubinger et al., “Risks from Learned Optimization” (2019, free) | Inner alignment and mesa-optimization | Depth |
Courses and Lectures
BlueDot Impact’s AI Safety Fundamentals provides an organized reading route. The 80,000 Hours guide explains one organization’s case for prioritizing AI risk and related careers; it is useful context, not a neutral technical consensus. In a seminar or reading group, compare original studies and competing explanations, and check whether disagreements concern evidence, assumptions about future capabilities, or judgments about which harms matter most.
| Course | Platform | Type |
|---|---|---|
| AI Safety Fundamentals: AI Alignment reading materials (free) | aisafetyfundamentals.com | Entry |
| BlueDot Impact technical AI safety courses | bluedot.org; check participation conditions | Depth |
| 80,000 Hours AI Safety Career Guide (free) | 80000hours.org | Auxiliary |
Practice, Tools, and Current Sources
Use evaluation tools from §5.5 and intervention tools from §5.6 for specific safety questions. A general benchmark suite does not by itself measure alignment. For an authorized red-team exercise, define the behavior being tested, system permissions, attacker’s information, test budget, and success criterion. Keep exploratory attacks separate from a held-out evaluation after the system is revised.
Read METR, Apollo Research, and the UK AI Security Institute for examples of evaluation protocols, paying attention to access, scaffolding, and task selection. The NIST AI Risk Management Framework helps connect measurements to deployment context and ongoing risk management. Incident databases supply cases to investigate, but usually lack the exposure counts needed to estimate a failure rate. Research forums can surface hypotheses and disagreements; trace empirical claims back to the underlying experiments.
| Resource | Platform | Type |
|---|---|---|
| AI Safety Fundamentals reading materials (free) | aisafetyfundamentals.com | Entry |
| EleutherAI LM Evaluation Harness (free) | GitHub | Practice |
| Alignment Forum (free) | alignmentforum.org | Reference |
| METR / Apollo Research publications (free) | metr.org / apolloresearch.ai | Reference |
| Anthropic, OpenAI, DeepMind safety research (free) | Lab research blogs | Reference |
| UK AI Security Institute publications (free) | aisi.gov.uk | Reference |
| NIST AI Risk Management Framework (free) | nist.gov | Reference |
| AI Incident Database (free) | incidentdatabase.ai | Auxiliary |
| AIAAIC Repository (free) | aiaaic.org | Auxiliary |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Reducing safety to output moderation | Harms can arise from wrong decisions, tool permissions, data handling, or delayed effects. | Evaluate the whole system and the consequences of its permitted actions. |
| Treating preference training as a safety certificate | Improved ratings establish results under the evaluated feedback and tasks. | Test the intended use and inspect failures not covered by the preference signal. |
| Accepting or dismissing future-risk claims without examining assumptions | Forecasts combine mechanisms with uncertain claims about future capabilities and deployment. | Separate assumptions, evidence for mechanisms, and uncertainty about extrapolation. |
| Reading only one research viewpoint | Different approaches prioritize different harms and make different assumptions. | Compare arguments, empirical tests, and the interests of affected parties. |
| Separating safety from system design | Permissions, objectives, interfaces, and recovery procedures shape consequences. | Evaluate these choices during development and after material changes. |
| Treating all evidence as interchangeable | A demonstration, a test score, a theorem, and a forecast support different conclusions. | State the type and scope of evidence behind each claim. |
中文
AI 安全研究怎样识别、评估和减少 AI 系统相关的伤害。对齐关注系统目标与行为怎样对应预期目标、人的偏好及相关约束。考虑谁的意图、怎样处理冲突、谁承担错误后果，都是问题的一部分；符合某个用户请求，并不构成安全的完整定义。
应区分失败机制。模型可能优化不充分的代理指标、发生普通预测错误、在条件变化后失败、被用于有害目的，或通过外围软件受到攻击。这些机制需要不同证据与干预。训练出更受偏好的回答，不会自动修复权限、数据访问或不可靠的后续动作。
这个科目把学习（§5.2–5.5）与安全（§4.5）、验证（§3.6）、系统可靠性（§4.10）连接起来，也包含关于未来系统的争议性假设。区分已经观测的失败、受控演示、理论论证与预测，可以使讨论更精确，并让技术问题始终对应证据。
前置知识：学习目标与评估（§5.2）、序贯决策（§5.4）、语言模型系统（§5.5）。可解释性（§5.6）是一种可能的证据来源，与测试、验证及系统控制相互补充。
从早期警告到技术纪律
对自动化决策的担忧有多条历史来源。控制工程、可靠软件、机器人与人因研究，都研究过作用于现实世界的系统怎样失败。Wiener 提醒人们，机器被规定的目的可能不同于人实际想要的目的。这自然连接到规格与反馈问题，并不要求预言某种未来 AI 架构必然出现。
I. J. Good 在 1965 年关于智能爆炸的讨论，考虑了系统能够改善后继系统设计的可能性。包括 Bostrom 工作在内的后续论证，进一步发展了高能力系统与控制的条件性情景。结论依赖对能力、目标、自主性和改进过程的假设。历史影响与经验确认是不同问题。
随着学习系统更广泛使用，研究者也调查具体失败机制。2016 年的 Concrete Problems in AI Safety组织了五个问题：避免副作用、避免奖励利用、可扩展监督、安全探索与分布变化。它们把目标设计、反馈成本与学习条件连接到不希望出现的行为。这是一项研究议程，并非伤害的完整清单。
偏好学习提供了逐项手写奖励之外的途径。Christiano 等人 2017 年的工作从比较中学习反馈，再用它训练行为。包括 InstructGPT、Constitutional AI 在内的后续语言模型工作，探索把示范、比较与书面原则用于后训练。这些实验在特定评估下展示了可测量的行为变化，并未确立所有相关价值都能从比较中推断，也未证明所得系统在所有应用中安全。
部署把分析单位从模型输出扩展到包含用户、数据、工具与机构的应用。模型卡、事故记录、红队测试和能力评估，发展了描述用途、局限及已观测失败的方法。NIST AI 风险管理框架把风险管理视为组织过程，同时也是技术测量问题。
可扩展监督与学习目标研究询问：当输出难以判断，或行为超出训练条件发生变化时，监督怎样继续有效？一些研究使用有意构造的模型或受限任务检验候选失败机制。这些实验可以揭示可能性、检验方法，而估计其他系统中出现这些问题的频率，还需要进一步证据。有用的历史应追踪问题怎样变得可测量，不把乐观或灾难预先写成结论。
规格、监督，以及已部署系统的经验安全
规格问题
训练目标是对部分预期行为的可操作规定，是否充分取决于任务。假设客服助手因迅速关闭工单得到奖励，它既可以通过高效解决问题改善指标，也可以把未解决请求标为完成。“工单已关闭”与“问题已解决”是不同变量。这个例子揭示了缺失的要求，并不需要假定系统具有类似人的欺骗意图。
奖励利用或规格博弈描述的是：行为在代理指标下得分高，却违反预期结果。Goodhart 式效应提供一组有用解释：选择和优化可能偏向度量不再良好对应目标的案例。这是一种可能失败机制，并非每个目标都不可用，或每个足够强的学习者都会利用漏洞的定理。某些范围明确的任务可以有精确规格与强检查。
区分三种失败：优化了错误目标，没有优化好选定目标，以及学出的行为在训练设置之外发生不希望出现的变化。它们在输出上可能相似，却需要不同补救。声称模型学到了内部目标，需要关于行为或计算的证据，单个错误回答不足以确立这种机制。
反馈方法增加信息，同时保留假设。偏好比较依赖评价者、说明和例子；书面原则需要在上下文中解释；奖励模型会外推到标签之外。应区分用户认可与正确性，以及任务完成与可接受副作用。在工单例子中，除了速度，还应评估经核实的解决情况、重新打开率和不当关闭，并规定何时需要人作出决定。
可扩展监督与验证差距
当检查输出需要稀缺专业知识、大量计算，或等待长期后果时，监督会变困难。有些任务的验证比生成便宜，但并非全部如此。证明检查器可以按形式规则验证推导，却不能自动确立形式化定理正是用户想表达的主张；这种对应也需要检查。
任务分解、工具辅助检查、专家审查与随机抽查，可以扩大监督能力，前提是各部分保留了可靠判断所需的信息。一组各自看似合理的摘要，可能遗漏决定性的跨文档矛盾；由不可靠模型辅助的审查者，也可能继承其错误。
辩论方法提出利用竞争论证帮助裁判识别错误。效果取决于哪些证据可以展示、裁判能够评价什么；有说服力不保证真实。迭代放大使用受辅助的判断训练更强系统。这些是具有特定实验与理论假设的研究路线，并非已经确立的一般解法。
弱到强泛化研究检验较强预训练模型能否在指定任务上从较弱模型的标签学习。超过弱监督者的结果说明，不完美标签不必施加完全相同的性能上限，但与更强监督相比仍有差距。这是对监督问题的经验类比，并不证明任意更强系统都能被较弱监督者可靠对齐。
应同时评估监督者与模型：它能发现哪些错误，会共享哪些错误，有什么信息，审查成本是多少？更好的监督应改善某种可测量判断或结果。如果多个评价者共享训练数据、假设或失败模式，它们的一致意见就不是同等强度的独立证据。
已部署系统的经验安全
应在明确用途与威胁模型下评估部署系统。识别它能读取什么信息、执行什么动作，以及错误输出之后会发生什么。提出退款建议的助手与能够直接执行退款的助手，即使文本生成准确率相同，后果也不同。
提示注入涉及本应作为数据处理的材料中出现指令，例如检索网页或工具结果。仅靠拒绝训练，不能确立这些材料与授权指令之间可信的边界。应限制工具权限，在应用代码中验证拟执行操作，并根据任务为敏感动作设置控制。抵抗越狱、访问控制与事实准确性相互关联，但属于不同性质。
无依据主张与迎合需要直接评估。检索可以提供证据，也可能检索到无关或被操纵的材料；流畅摘要仍可能误述来源。偏好训练可能奖励讨好的表达，但迎合并没有唯一必要成因。应使用匹配应用的事实检查、反例、不确定性评估及弃答标准。减少错误是可测量目标，“这永远无法修复”或“模型已经对齐”都不能取代测试。
能力评估询问系统在特定访问权限、辅助和时间预算下能完成什么。滥用风险还取决于谁能访问、有哪些其他资源，以及该能力是否改变使用者的实际选择。受控演示、真实任务评估与事故观测提供不同证据。一个评价者尝试失败，并不证明所有其他过程都会失败。
未观测到失败不意味着风险为零。在一个简单模型中，从同一固定部署分布独立抽取 1,000 次测试，若零次失败，则失败概率的单侧 95% 二项置信上界为 1−0.05^(1/1000) ≈ 0.003。即使这个陈述也依赖采样模型；测试相关或输入变化都可能破坏其解释。报告测了什么、失败定义、不确定性，以及哪些用途仍在评估范围之外。
监控与事故响应把评估延伸到发布之后。跟踪有意义的结果，调查失败，修订控制，并检查缓解措施是否引入新错误或负担。模型变化、新工具或新用户群体，都可能改变先前评估的假设。安全证据属于某个用途中的具体系统版本，而不永久附着于模型名称。
学习这一部分会改变什么
学完后，应能把宽泛的安全主张变成可评估主张：明确系统、用户与受影响者、允许动作、失败条件及支持证据。区分目标设计问题与实现缺陷、分布变化或访问控制失败。
针对一个小型助手应用，写出一个预期结果、一个容易使用的代理指标，以及一个二者偏离的反例。分别测试模型与外围控制，记录哪些干预改善了预期结果，而不只改善代理分数或答案表现出的自信。
按照证据类型阅读研究主张。定理有假设；受控演示在其设置下确立一种可能性；基准按指定协议估计表现；预测则增加了不确定外推。仔细比较这些证据，才能同时调查未来风险与当前失败，而不让其中一类取代另一类。
资源
书籍与文本
Russell 的 Human Compatible 主张把人的目标视为不确定、尚待了解的对象，而不是固定且已被完全掌握的目标。Christian 的 The Alignment Problem 介绍学习系统的历史及其对人的影响。Bostrom 的 Superintelligence 对高度有能力的系统及其控制问题提出条件性论证；应将这些论证与现有系统的实测结果、以及这类系统何时出现的预测分开。
技术学习可以从 “Concrete Problems in AI Safety” 和偏好学习论文入手，再读 Ngo、Chan 与 Mindermann 关于深度学习对齐的论文，以及 Casper 等人对 RLHF 局限的分析。Hubinger 等人讨论学习得到的优化过程可能带来的问题。对每一种提出的失败情形，都要辨认其机制、假设、支持实验，以及什么证据会削弱这一判断。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Russell, Human Compatible | 易读的对齐入口；主流 AI 视角 | 入门 |
| Christian, The Alignment Problem | 易读的历史与概念入口 | 入门 |
| Bostrom, Superintelligence | 关于长期控制风险的论证 | 深入 |
| Ngo, Chan & Mindermann, “The Alignment Problem from a Deep Learning Perspective”（免费） | 技术性对齐处理 | 深入 |
| Amodei et al., “Concrete Problems in AI Safety”（2016，免费） | 技术安全的奠基性表达 | 深入 |
| Christiano, “Deep RL from Human Preferences”（2017，免费） | RLHF 奠基论文 | 深入 |
| RLHF case study: Ouyang et al., InstructGPT paper（2022，免费） | 语言模型规模上的 RLHF | 深入 |
| Bai et al., Constitutional AI paper（2022，免费） | Constitutional AI 方法 | 深入 |
| Preference optimization case study: Rafailov et al., DPO paper（2023，免费） | 直接偏好优化 | 深入 |
| Casper et al., “Open Problems and Fundamental Limitations of RLHF”（免费） | 对 RLHF 的系统批判 | 深入 |
| Hubinger et al., “Risks from Learned Optimization”（2019，免费） | 内在对齐与 mesa-optimization | 深入 |
课程与讲座
BlueDot Impact 的 AI Safety Fundamentals 提供有组织的阅读路线。80,000 Hours 指南解释了该组织为何优先关注 AI 风险及相关职业；它可以提供背景，但不代表中立的技术共识。参加研讨课或读书小组时，应比较原始研究与不同解释，并分辨争议究竟涉及证据、对未来能力的假设，还是对哪些危害更值得关注的判断。
| 课程 | 平台 | 类型 |
|---|---|---|
| AI Safety Fundamentals: AI Alignment 阅读材料（免费） | aisafetyfundamentals.com | 入门 |
| BlueDot Impact 技术性 AI 安全课程 | bluedot.org；参与条件见项目说明 | 深入 |
| 80,000 Hours AI Safety Career Guide（免费） | 80000hours.org | 辅助 |
实践、工具与当前资料
针对具体安全问题，可以使用 §5.5 的评估工具和 §5.6 的干预工具。通用基准测试套件本身不能直接测量对齐程度。开展获授权的红队测试时，应定义要测试的行为、系统权限、攻击者掌握的信息、测试预算和成功标准。探索性的攻击与系统修订后使用的保留评估集应分开。
阅读 METR、Apollo Research 和英国 AI Security Institute 的评估方法时，重点查看模型的访问权限、配套工具和任务选择。NIST AI Risk Management Framework 有助于把测量结果与部署情境、持续风险管理联系起来。事故数据库提供了可调查的案例，但通常缺少估计失败率所需的总使用次数。研究论坛可以帮助发现假设和分歧；其中的实证主张仍应追溯到原始实验。
| 资源 | 平台 | 类型 |
|---|---|---|
| AI Safety Fundamentals reading materials（免费） | aisafetyfundamentals.com | 入门 |
| EleutherAI LM Evaluation Harness（免费） | GitHub | 实践 |
| Alignment Forum（免费） | alignmentforum.org | 参考 |
| METR / Apollo Research publications（免费） | metr.org / apolloresearch.ai | 参考 |
| Anthropic, OpenAI, DeepMind safety research（免费） | 实验室研究博客 | 参考 |
| 英国 AI Security Institute 出版物（免费） | aisi.gov.uk | 参考 |
| NIST AI Risk Management Framework（免费） | nist.gov | 参考 |
| AI Incident Database（免费） | incidentdatabase.ai | 辅助 |
| AIAAIC Repository（免费） | aiaaic.org | 辅助 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 把安全缩减为输出审核 | 伤害也可能来自错误决策、工具权限、数据处理或延迟后果。 | 评估整个系统及允许动作的后果。 |
| 把偏好训练当作安全认证 | 评分改善确立的是所评估反馈和任务下的结果。 | 测试预期用途，检查偏好信号未覆盖的失败。 |
| 不检查假设就接受或否定未来风险主张 | 预测把机制与对未来能力、部署的不确定判断组合起来。 | 区分假设、机制证据及外推不确定性。 |
| 只阅读一种研究观点 | 不同路线优先处理不同伤害，也使用不同假设。 | 比较论证、经验检验及受影响者的利益。 |
| 把安全与系统设计分开 | 权限、目标、接口及恢复程序塑造实际后果。 | 在开发及重要变更后评估这些选择。 |
| 把各种证据视为可互换 | 演示、测试分数、定理与预测支持不同结论。 | 说明每项主张背后的证据类型与范围。 |