English
Interpretability asks for understandable, testable accounts of a model’s behavior. Different questions require different methods: which inputs affect a prediction, what information can be recovered from a representation, which internal computations contribute to an output, and whether an explanation helps a person detect errors. A method useful for one question may not answer the others.
Mechanistic interpretability investigates the computations inside a learned model. It forms hypotheses about representations and pathways, then tests them with measurements and interventions. The goal is to explain specified behavior at a useful level of detail. A meaningful feature label or an attractive visualization is a starting hypothesis, not a complete account of the computation.
Internal analysis can complement behavioral testing by locating a failure mechanism or predicting a new failure case. It does not automatically establish reliability on every input. An understandable model can still implement an unsuitable objective, and some properties of a complicated model can be checked without understanding every learned feature. The relevant question is what claim the available evidence supports.
Prerequisites: Neural-network computation and gradients (§5.3), basic experimental evaluation (§5.2), and transformer structure (§5.5) for the language-model examples. Begin with a small model whose activations and outputs can be inspected directly.
From Feature Visualization to Mechanistic Understanding
Explaining a model did not begin with recent transformer research. Statistical modeling, rule systems, decision trees, sensitivity analysis, and neural-network visualization supplied different traditions. Expert systems could show a rule trace, but the trace still depended on the adequacy of the encoded knowledge. Learned models raised an additional question: how can their acquired representations and computations be described after training?
Feature visualization searches for inputs that activate a unit or direction, often by optimizing an input while regularizing it toward interpretable patterns. Work summarized in Distill’s 2017 articles made the method and its choices accessible. Optimized images and naturally occurring high-activation examples provide complementary evidence; either can be misleading if the analysis selects only examples fitting a preferred interpretation.
The Circuits work in 2020 connected feature observations to compositions of computations in convolutional networks. The 2021 transformer-circuits framework analyzed how attention heads and other components read and write representations in the residual stream. This supplied explicit algebra and candidate mechanisms for studying particular architectures. It did not establish that every model uses the same set of human-named components.
The 2022 induction-head study investigated a pattern-completion mechanism associated with in-context learning. The authors distinguished strong causal evidence in small attention-only models from correlational evidence in larger models with MLPs. That distinction matters: explaining one pattern-completion behavior does not establish one mechanism for all learning from context.
Superposition research examined how several features may be represented by overlapping directions. Sparse autoencoders then offered a way to fit a sparse dictionary to recorded activations. The 2023 monosemanticity work and 2024 experiments on Claude 3 Sonnet found interpretable candidate features and tested some by steering activations. These results expanded the scale of investigation while leaving questions about completeness, interpretation, and side effects.
The 2025 Circuit Tracing work used more interpretable replacement components to build attribution graphs and investigate selected behaviors, including in a deployed model. The replacement is an approximation whose faithfulness must be evaluated. It would be inaccurate to say that circuit analysis is confined to tiny models, or that these advances amount to a complete explanation of a production model.
Features, Circuits, and the Limits of What Can Be Known
What Features Models Represent
A feature is a proposed property represented by some aspect of a model’s state. Many methods look for linear directions in an activation space because they are tractable and often useful. This is a modeling hypothesis, not a theorem that every concept is linear, localized, or aligned with a single neuron. Features may be distributed, nonlinear, or dependent on context.
Polysemanticity means that one unit responds in several apparently different contexts. Superposition is a possible representational mechanism: multiple features use overlapping directions, allowing economical representation when only a few are active together, at a cost of interference. Polysemantic observations alone do not prove that a particular superposition model explains them.
A sparse autoencoder approximates an activation vector with a ≈ Dz + b, where the columns of D form a learned dictionary and z has relatively few active coefficients. One objective combines reconstruction error with a sparsity penalty, such as ‖a−(Dz+b)‖² + λ‖z‖₁. Dictionary size, sparsity, training data, and optimization influence the decomposition. A feature may split across dictionary elements, and some behaviorally important information may remain in the reconstruction error.
Assess candidate features using diverse positive and negative examples, held-out activations, and interventions. A probe that predicts a property from activations shows that information is recoverable by that probe; it does not establish that the original model uses it for the output under study. Steering a direction can show an effect, but changes in unrelated outputs or unnatural activation states may complicate the interpretation. Automated labels for features are hypotheses to evaluate.
Circuits: How Features Compose
A circuit is a proposed subset of computations contributing to a specified behavior. Defining the behavior comes first: repeating a token pattern, resolving a name in a sentence, or selecting one answer over another. The same model can use different pathways on different inputs, and several pathways may redundantly support one behavior.
A simple induction mechanism completes [A][B] … [A] with [B]. In a two-layer construction, a previous-token head can place information about [A] at the position containing [B]. At the later [A], another head can match that previous-token information and copy information about [B]. This explains a particular way to reuse an observed association. It does not establish that all in-context tasks reduce to copying or that every large model implements this exact construction.
Activation patching tests such hypotheses by replacing selected internal activations from one run with those from another. Consider sentences differing only in whether Alice gives a book to Bob or to Carol, followed by a question about the recipient. If patching selected activations restores preference for Bob in the Carol run, that is evidence that the patched computation can mediate this output difference. It does not prove that the fact is stored permanently in that one location or that the pathway is uniquely necessary.
Use matched prompts, align token positions, specify the output metric, and compare interventions at unrelated locations. Zero ablation, mean ablation, and replacement with another input’s activations create different conditions. An intervention may push the model away from naturally occurring states. Test the proposed circuit on new cases and examine both what its removal disrupts and what its retained computation can reproduce. A plausible graph becomes an explanation through these checks.
What Can and Cannot Be Verified
Separate observation, prediction, intervention, and generalization. A component correlating with a behavior is an observation. Predicting how an intervention will change that behavior is a stronger hypothesis. Confirming the prediction supports a causal claim within the experiment. Testing new inputs assesses how far the account extends. None of these steps automatically supplies a global safety guarantee.
Gradient saliency measures local sensitivity under a chosen parameterization; it need not describe the effect of a large input change. Attention weights describe one part of information routing, while value vectors and other paths also affect the output. A language model’s written explanation is itself an output, not a privileged readout of the computation that produced the answer. Each can be useful evidence when paired with an appropriate test.
For dictionary or replacement-model methods, measure reconstruction quality and downstream behavioral fidelity as well as interpretability of the labels. Explain which components remain unanalyzed and how much of the target behavior is recovered. Simpler surrogate explanations may omit interactions that matter outside the tested examples.
Keep the scientific claim narrow enough to challenge: which model checkpoint, which behavior, which input family, which interventions, and which competing explanations? A failure on a new case can improve the hypothesis. An account that explains every possible outcome after the fact is difficult to test and offers little predictive value.
What Studying This Changes
You should be able to state what an interpretability experiment is trying to explain and what evidence would count against the explanation. Distinguish information recoverable from a representation from information the model demonstrably uses in a tested computation.
Begin with a small repeated-token task or a controlled name-resolution task. Record the baseline behavior, formulate a circuit hypothesis, run interventions and controls, then evaluate new examples. Preserve failed interventions as well as successful visualizations; they constrain which explanation is credible.
Use internal analysis together with ordinary evaluation. A proposed mechanism can suggest new tests, and behavioral failures can guide where to inspect the model. Interpretability is most useful when it improves a concrete prediction or diagnosis, rather than supplying a reassuring label for an otherwise untested system.
Resources
Books and Survey Articles
Molnar’s Interpretable Machine Learning introduces the broader field, including feature importance and explanations of individual predictions. Distill’s Feature Visualization and Circuits articles show how visual examples and interventions can support hypotheses about neural computation. Keep the distinction between describing a feature and showing how the model uses it.
For transformers, read “A Mathematical Framework for Transformer Circuits” after you can follow an attention layer in code. Study the induction-head and superposition papers with their toy experiments, then examine what changes when sparse autoencoders are applied to larger models. Lipton’s “The Mythos of Model Interpretability” helps clarify what different authors mean by an explanation. None of these methods should be assessed only by how convincing its visualization looks.
| Resource | Role | Type |
|---|---|---|
| Molnar, Interpretable Machine Learning (3rd ed., free) | Broad interpretability landscape | Entry |
| Distill Feature Visualization + Circuits series (free) | Feature visualization and circuit analysis | Practice |
| Elhage et al., “Mathematical Framework for Transformer Circuits” (2021, free) | Transformer interpretability foundation | Depth |
| Olsson et al., “In-context Learning and Induction Heads” (2022, free) | Foundational circuit discovery | Depth |
| Elhage et al., “Toy Models of Superposition” (2022, free) | Polysemanticity foundation | Depth |
| Bricken et al., “Towards Monosemanticity” (2023, free) | Sparse autoencoder foundation | Depth |
| Templeton et al., “Scaling Monosemanticity” (2024, free) | SAE at production scale | Depth |
| Meng et al., ROME paper (2022, free) | Factual association and patching | Depth |
| Nanda, “Concrete Steps for Mechanistic Interpretability” (free) | Accessible practical entry | Entry |
| Lipton, “The Mythos of Model Interpretability” (2018, free) | Clarifying the goals of explanations | Depth |
Courses and Lectures
The ARENA curriculum provides publicly available programming exercises on transformers and mechanistic interpretability. Neel Nanda’s tutorials and lectures provide additional walkthroughs. These are starting sequences, not prerequisites that every learner must finish. BlueDot Impact’s alignment reading materials supply safety context; distinguish access to readings from admission or participation in a taught program.
| Course | Platform | Type |
|---|---|---|
| ARENA mechanistic interpretability tutorials (free) | arena.education | Practice |
| Nanda YouTube lectures (free) | YouTube | Entry |
| AI Safety Fundamentals alignment reading materials (free) | aisafetyfundamentals.com | Entry |
Practice, Tools, and Current Sources
TransformerLens provides tools for inspecting and intervening on activations; SAELens supports sparse-autoencoder experiments. Neuroscope and Neuronpedia let you inspect activation examples. Always check the model, layer, dataset, and feature definition associated with an example before interpreting it.
A manageable project is to reproduce an induction pattern in a small attention-only transformer. Compare clean and perturbed inputs, test candidate heads by patching and ablation, and include controls that leave the hypothesized mechanism unchanged. If an intervention fails to change behavior, investigate whether the hypothesis was wrong, the intervention was ineffective, or another route compensated. Reproduction is an investigation, not a requirement to obtain the paper’s conclusion on any model.
| Resource | Platform | Type |
|---|---|---|
| TransformerLens (free) | pip install transformer-lens |
Practice |
| SAELens (free) | GitHub / pip | Practice |
| Neuroscope (free) | neuroscope.io | Practice |
| Neuronpedia (free) | neuronpedia.org | Practice |
| Gemma Scope (free) | deepmind.google | Reference |
| Anthropic Transformer Circuits Thread (free, ongoing) | transformer-circuits.pub | Reference |
| ARENA tutorial notebooks (free) | arena.education | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Equating a saliency map with a mechanism | Local sensitivity does not uniquely identify a computation or explain large changes. | Specify the claim and compare it with perturbations and internal interventions. |
| Treating a plausible feature label as verified | Selected examples can support several competing interpretations. | Use held-out positive and negative cases and predict intervention effects. |
| Assuming one neuron equals one concept | Representations can be distributed, overlapping, or context-dependent. | Compare individual units with candidate directions and alternative decompositions. |
| Treating SAE features as a complete dictionary of the model | Reconstruction error, feature splitting, and training choices affect what is found. | Measure behavioral fidelity and inspect unexplained residual information. |
| Skipping intervention controls | A large effect may arise from unnatural activations or unrelated disruption. | Compare matched interventions, alternative baselines, and new input families. |
| Generalizing a local circuit explanation to overall safety | The model can use other pathways or fail for reasons outside the analyzed behavior. | State coverage and combine internal evidence with system-level evaluation. |
中文
可解释性研究为模型行为提供可理解、可检验的说明。不同问题需要不同方法：哪些输入影响预测，能够从表示中恢复什么信息，哪些内部计算参与产生输出，以及解释能否帮助人发现错误。适合其中一个问题的方法，未必能回答其他问题。
机制可解释性研究学习模型内部的计算。它提出关于表示与通路的假设，再通过测量和干预检验，目标是在有用的细节层次上解释指定行为。一个有意义的特征标签或好看的可视化，只是起始假设，不是对计算的完整说明。
内部分析可以定位失败机制或预测新失败案例，从而补充行为测试，但不会自动确立模型在所有输入上的可靠性。可理解的模型仍可能实现不合适的目标；复杂模型的部分性质，也可以在未理解每个学习特征时检查。关键问题是现有证据支持什么主张。
前置知识：神经网络计算与梯度（§5.3）、基本实验评估（§5.2）；语言模型例子还需要 transformer 结构（§5.5）。先从能直接检查激活与输出的小模型开始。
从特征可视化到机制理解
解释模型并非始于近期 transformer 研究。统计建模、规则系统、决策树、敏感性分析与神经网络可视化提供了不同传统。专家系统可以展示规则轨迹，但轨迹仍依赖编码知识是否充分。学习模型进一步提出了一个问题：训练之后，怎样描述它获得的表示与计算？
特征可视化寻找能激活某个单元或方向的输入，常通过优化输入，同时用正则化使模式更容易解释。Distill 在 2017 年归纳的工作使方法及其选择更容易理解。优化生成的图像与自然出现的高激活例子提供互补证据；如果只挑选符合预期解释的例子，两者都可能误导。
2020 年的 Circuits 工作把特征观测连接到卷积网络中计算的组合；2021 年的 transformer circuits 框架分析注意力头及其他组件怎样在残差流中读写表示。这为研究特定架构提供了明确代数与候选机制，并未确立所有模型都使用同一套由人命名的组件。
2022 年的归纳头研究考察了与上下文学习相关的一种模式补全机制。作者区分了小型纯注意力模型中的较强因果证据，以及带 MLP 的较大模型中的相关证据。这个区别很重要：解释一种模式补全行为，不等于确立所有上下文学习都使用同一个机制。
叠加研究考察多个特征怎样由重叠方向表示。稀疏自编码器随后提供了对记录激活拟合稀疏字典的方法。2023 年的单义性研究，以及 2024 年 Claude 3 Sonnet 实验，发现了可解释的候选特征，并通过激活引导检验其中一些。这些结果扩大了研究规模，但仍留下完整性、解释与副作用问题。
2025 年的 Circuit Tracing 工作使用更易解释的替代组件构建归因图，调查包括已部署模型在内的特定行为。替代组件是一种近似，需要评估其忠实程度。因此，说电路分析只限于极小模型并不准确；把这些进展等同于完整解释生产模型，也不准确。
特征、回路，以及可知性的边界
模型表示了什么特征
特征是关于模型状态某一部分表示了什么性质的假设。许多方法寻找激活空间中的线性方向，因为这容易处理且经常有用。但它是一种建模假设，并非每个概念都线性、局部化或对应单个神经元的定理。特征可以分布式、非线性，也可以依赖上下文。
多义性指同一个单元在几种看起来不同的上下文中响应。叠加是一种可能的表示机制：多个特征使用重叠方向，在同时只有少数特征激活时节省表示成本，代价是相互干扰。仅观测到多义性，并不证明某个具体叠加模型能够解释它。
稀疏自编码器用 a ≈ Dz + b 近似激活向量，其中 D 的列形成学习字典，z 只有较少活跃系数。一种目标把重建误差与稀疏惩罚组合起来，例如 ‖a−(Dz+b)‖² + λ‖z‖₁。字典大小、稀疏程度、训练数据与优化都会影响分解。一个特征可能被拆分到多个字典元素中，某些对行为重要的信息也可能留在重建误差里。
应使用多样的正反例、保留激活样本和干预来评估候选特征。探针能够从激活预测某个性质，说明这个探针可以恢复相关信息，并不确立原模型在产生所研究输出时使用了它。引导某个方向可以展示作用，但无关输出的变化或不自然的激活状态可能使解释复杂化。自动产生的特征标签也是待检验假设。
回路：特征如何组合
电路是被假定参与指定行为的一组计算。首先要定义行为：重复 token 模式、解析句中名字，或在两个答案之间作出选择。同一模型可能对不同输入使用不同通路，多条通路也可能冗余地支持一种行为。
一个简单归纳机制会把 [A][B] … [A] 补成 [B]。在两层构造中，前一 token 头可以把关于 [A] 的信息放到包含 [B] 的位置；遇到后来的 [A] 时，另一个头匹配此前 token 信息，并复制关于 [B] 的信息。这解释了一种复用已观测关联的方法，并不确立所有上下文任务都能归结为复制，也不确立每个大模型都采用这个精确构造。
激活修补通过把一次运行的选定内部激活替换为另一次运行的激活，来检验这类假设。考虑只在 Alice 把书给 Bob 还是 Carol 上不同的句子，然后询问收书人。如果在 Carol 那次运行中修补选定激活，恢复了对 Bob 的偏好，就说明被修补的计算能够影响这个输出差异。这不证明事实永久存储于该处，也不证明这条通路是唯一必要通路。
使用匹配提示，对齐 token 位置，明确输出指标，并比较无关位置的干预。置零消融、均值消融、用其他输入激活替换，创造的是不同条件。干预还可能使模型偏离自然出现的状态。应在新案例上检验候选电路，同时考察删除它会破坏什么，以及保留它能复现什么。通过这些检查，一个看似合理的图才逐渐成为解释。
什么可以被验证，什么不能
区分观测、预测、干预与泛化。组件与行为相关是一项观测；预测干预会怎样改变行为，是更强的假设；预测被确认，支持实验范围内的因果主张；在新输入上测试，考察解释能够扩展多远。这些步骤都不自动提供全局安全保证。
梯度显著性衡量选定参数化下的局部敏感性，不一定描述大幅输入变化的效果。注意力权重描述信息路由的一部分，值向量与其他路径同样影响输出。语言模型写出的解释本身也是输出，不是直接读取产生答案的内部计算。在配合恰当测试时，这些材料都可以成为有用证据。
对于字典或替代模型方法，除了标签是否可理解，还应测量重建质量与下游行为忠实度。说明哪些组件尚未分析，以及恢复了目标行为的多少。更简单的代理解释可能遗漏在测试例子之外重要的相互作用。
让科学主张明确到可以质疑：哪个模型检查点、哪种行为、哪类输入、哪些干预，以及哪些竞争解释？新案例中的失败可以帮助改善假设。如果一种说法事后能解释所有可能结果，就很难检验，也没有多少预测价值。
学习这一部分会改变什么
学完后，应能陈述可解释性实验试图解释什么，以及什么证据会反对该解释。区分可以从表示恢复的信息，与模型在被测计算中确实使用的信息。
从小型重复 token 任务或受控名字解析任务开始。记录基线行为，提出电路假设，进行干预和对照，再评估新例子。除了成功的可视化，还应保留失败干预；它们限制了哪些解释可信。
把内部分析与普通评估结合起来。候选机制可以提示新测试，行为失败可以指导检查模型的哪个部分。当可解释性改善了具体预测或诊断时，它更有用；给未充分测试的系统贴上让人安心的标签，并不增加这种证据。
资源
书籍与综述文章
Molnar 的 Interpretable Machine Learning 介绍较广义的可解释性，包括特征重要性和单次预测的解释。Distill 的 Feature Visualization 与 Circuits 系列展示如何用可视化实例和干预支持关于神经网络计算的假设。阅读时要区分：描述一个特征，与说明模型如何使用这个特征。
能够跟踪注意力层的代码后，再读 “A Mathematical Framework for Transformer Circuits”。结合小型实验学习归纳头和叠加表示论文，再考察稀疏自编码器用于更大模型时发生了什么变化。Lipton 的 “The Mythos of Model Interpretability” 有助于辨明不同作者所说的“解释”具体指什么。评价这些方法时，不能只看可视化结果是否令人信服。
| 资源 | 作用 | 类型 |
|---|---|---|
| Molnar, Interpretable Machine Learning（第 3 版，免费） | 广义可解释性图景 | 入门 |
| Distill Feature Visualization + Circuits series（免费） | 特征可视化与回路分析 | 实践 |
| Elhage et al., “Mathematical Framework for Transformer Circuits”（2021，免费） | Transformer 可解释性基础 | 深入 |
| Olsson et al., “In-context Learning and Induction Heads”（2022，免费） | 基础回路发现 | 深入 |
| Elhage et al., “Toy Models of Superposition”（2022，免费） | 多义性基础 | 深入 |
| Bricken et al., “Towards Monosemanticity”（2023，免费） | 稀疏自编码器基础 | 深入 |
| Templeton et al., “Scaling Monosemanticity”（2024，免费） | 生产规模 SAE | 深入 |
| Meng et al., ROME paper（2022，免费） | 事实关联与 patching | 深入 |
| Nanda, “Concrete Steps for Mechanistic Interpretability”（免费） | 易上手的实践入口 | 入门 |
| Lipton, “The Mythos of Model Interpretability”（2018，免费） | 澄清解释的目标 | 深入 |
课程与讲座
ARENA 课程材料 提供公开的 Transformer 和机制可解释性编程练习。Neel Nanda 的教程和讲座提供补充讲解。这些是可选的学习顺序，不是人人都必须完成的前置要求。BlueDot Impact 的对齐阅读材料可以补充安全背景；访问阅读材料与获准参加授课项目是不同的事。
| 课程 | 平台 | 类型 |
|---|---|---|
| ARENA mechanistic interpretability tutorials（免费） | arena.education | 实践 |
| Nanda YouTube lectures（免费） | YouTube | 入门 |
| AI Safety Fundamentals 对齐阅读材料（免费） | aisafetyfundamentals.com | 入门 |
实践、工具与当前资料
TransformerLens 提供检查和干预激活的工具，SAELens 支持稀疏自编码器实验。Neuroscope 和 Neuronpedia 可用于查看激活实例。解释一个实例之前，先核对它对应的模型、层、数据集和特征定义。
一个可控制规模的项目，是在小型纯注意力 Transformer 中复现归纳模式。比较原始输入与扰动输入，用激活替换和消融检验候选注意力头，并加入不改变所假设机制的对照。如果干预没有改变行为，需要检查是假设错误、干预没有生效，还是其他路径起到了补偿作用。复现是一项调查，不是要求任意模型都产生论文中的结论。
| 资源 | 平台 | 类型 |
|---|---|---|
| TransformerLens（免费） | pip install transformer-lens |
实践 |
| SAELens（免费） | GitHub / pip | 实践 |
| Neuroscope（免费） | neuroscope.io | 实践 |
| Neuronpedia（免费） | neuronpedia.org | 实践 |
| Gemma Scope（免费） | deepmind.google | 参考 |
| Anthropic Transformer Circuits Thread（免费，持续更新） | transformer-circuits.pub | 参考 |
| ARENA tutorial notebooks（免费） | arena.education | 实践 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 把显著图等同于机制 | 局部敏感性不唯一确定计算，也不解释大幅变化。 | 明确主张，用扰动与内部干预检验。 |
| 把合理的特征标签当作已验证 | 挑选的例子可能支持多种竞争解释。 | 使用保留的正反例，并预测干预效果。 |
| 假定一个神经元对应一个概念 | 表示可能分布式、重叠或依赖上下文。 | 比较单元、候选方向及不同分解。 |
| 把 SAE 特征当作模型的完整字典 | 重建误差、特征拆分与训练选择影响发现结果。 | 测量行为忠实度，检查未解释的残余信息。 |
| 跳过干预对照 | 大幅效果可能来自不自然激活或无关破坏。 | 比较匹配干预、不同基线与新输入类别。 |
| 从局部电路解释推到整体安全 | 模型可能使用其他通路，也可能因分析范围外原因失败。 | 说明覆盖范围，把内部证据与系统评估结合。 |