English
A neural network is a function. Given input, it produces output. Training adjusts its parameters until the output matches a target well enough. This description is complete and tells you almost nothing about what the function is doing internally — which computations it is performing, what concepts it is tracking, how intermediate layers process information on its way from input to output. Interpretability is the discipline of answering those questions: not what the network outputs, but what it is doing.
The broader interpretability literature includes techniques like attention visualization and saliency maps, which highlight which parts of an input most affect the output. These are useful but limited: they describe the relationship between input and output without characterizing the internal computation. Mechanistic interpretability goes deeper, aiming to reverse-engineer the actual computational circuits within a network — to identify what features individual neurons detect, what circuits they form to compute specific functions, and how those circuits compose into the model’s overall behavior. The goal is understanding that survives causal scrutiny, not patterns that look explanatory but may not reflect how the model actually operates.
This discipline matters for AI safety in a direct and structural way. A model whose internal mechanisms are understood is a model whose behavior under novel conditions, adversarial inputs, or capability expansions can be reasoned about. A model that is a black box can only be tested; it cannot be guaranteed. As language models are deployed in high-stakes settings — medical, legal, security — the gap between “tested extensively” and “mechanisms understood” becomes an engineering gap with real consequences.
Prerequisites: Deep learning (§5.3) — transformer architecture in detail, attention mechanisms, residual streams. Machine learning foundations (§5.2) — linear algebra at the level of understanding how activations are linear combinations of features. Large language models (§5.5) — the models that most contemporary interpretability targets.
From Feature Visualization to Mechanistic Understanding
Interpretability research has a surprisingly short history as a systematic empirical discipline. The ingredients — trained neural networks with interesting internal structure — have existed since the early deep learning era. The systematic project of understanding that structure is largely a product of the last decade, and most of the core conceptual vocabulary was established between 2017 and 2023.
The founding work is a set of papers by Chris Olah and colleagues, culminating in his 2017 Distill article “Feature Visualization.” The central observation was simple: you can understand what a neuron detects by finding the input that maximally activates it — by running gradient ascent on the input to maximize the neuron’s activation. This produced interpretable images for convolutional network neurons: neurons in early layers maximally activated by edges and gabor patterns; neurons in later layers maximally activated by faces, animals, or specific textures. The technique was not new — DeepDream had used similar gradient ascent in 2015 — but the systematic, careful application to understand what individual neurons were computing was new. The Distill format, with its interactive visualizations, made the findings unusually accessible.
The Circuits thread (Olah, Cammarata, Schubert, et al., 2020) extended feature visualization from individual neurons to the connections between them. The key insight was that neurons are not isolated; they form circuits: compositions of features connected by weights that implement specific computational functions. The paper identified specific circuits in convolutional networks — curve detectors, high-low frequency detectors, multi-frequency detectors — and traced the computations they performed step by step. The approach was explicitly neuroscience-inspired: “circuit” borrowed the neuroscience vocabulary of local circuits performing specific functions within a larger brain region. Whether the analogy to biological circuits holds in deep ways is contested, but the approach of identifying functional units and characterizing their computations proved productive.
The shift from convolutional networks to transformers required new methods. Elhage et al.’s “A Mathematical Framework for Transformer Circuits” (Anthropic, 2021) established the conceptual vocabulary for transformer interpretability: the residual stream as a communication channel that each layer reads from and writes to, attention heads as low-rank operations on the residual stream, MLP layers as key-value memories. The paper described specific computations that could be performed by one-layer and two-layer transformers, providing the first systematic framework for thinking about what transformers are computing rather than only what they output.
The induction heads paper (Olsson, Elhage, Nanda et al., 2022) was the first discovery of a specific circuit in a production-scale language model. Induction heads are a two-head circuit that enables in-context learning: one attention head copies previous token patterns to the current context, and a second head attends to positions that previously followed the same token. This circuit implements the basic mechanism behind in-context learning — if you have seen “[A] [B] … [A]” before, predict “[B]” — and the paper demonstrated that this circuit forms during training, appears across architectures, and causally mediates the model’s ability to complete patterns. The causal verification — showing that disrupting the circuit degrades performance in exactly the predicted way — was a methodological advance: not just “this circuit is there” but “this circuit is doing this.”
The polysemanticity problem disrupted simple feature analysis. A clean story would be that individual neurons represent individual features — one neuron for “smile,” one for “dog,” one for “past tense.” The reality, documented progressively, is that most neurons are polysemantic: they respond to multiple unrelated features simultaneously. This is not noise; it reflects a mathematical fact about how networks store more features than they have neurons, by exploiting the high-dimensional geometry to keep features nearly orthogonal even when many more features than neurons are represented. Elhage et al.’s “Toy Models of Superposition” (2022) provided the first systematic theoretical and empirical treatment: showing when superposition occurs, why it is advantageous, and what the resulting representational geometry looks like.
Polysemanticity made neuron-level analysis fundamentally limited. If each neuron encodes multiple features, then understanding the neuron does not give you the features. The response was sparse autoencoders (SAEs): train an auxiliary network that takes activations as input and decomposes them into sparse combinations of learned directions, each of which corresponds to a single interpretable feature. Bricken et al.’s “Towards Monosemanticity” (2023) demonstrated that SAEs trained on a single-layer transformer in a toy setting produced thousands of interpretable, monosemantic features. Templeton et al.’s “Scaling Monosemanticity” (2024) scaled this to Claude 3 Sonnet, finding millions of interpretable features including abstract concepts: the “Golden Gate Bridge,” “banana,” “Martin Luther King,” and “the concept of sycophancy” were all identifiable as coherent directions in activation space. Artificially activating the “banana” feature caused the model to talk about bananas unprompted; activating the “sycophancy” feature changed how the model balanced agreement with users against accuracy. These were causal demonstrations — features that mechanically influenced model behavior when directly stimulated.
The current frontier is the investigation of whether features can be composed into higher-level circuits, whether circuits can be identified automatically at scale (circuit discovery has been largely manual and labor-intensive), and whether interpretability methods can reveal safety-relevant internal structure: whether a model that appears helpful is computing representations consistent with that, or whether concerning representations might be hidden under the surface behavior. The intersection of interpretability with safety is where most of the applied motivation originates.
Features, Circuits, and the Limits of What Can Be Known
What Features Models Represent
The feature is the fundamental unit of mechanistic interpretability. A feature is a direction in activation space — a linear combination of neuron activations — that corresponds to a humanly-interpretable concept. The key result is that features are linear: whether a model is “thinking about” a particular concept corresponds to whether the corresponding direction in its activation space has high component weight.
The evidence for linearity comes from multiple directions. Word embeddings famously show linear geometry: “king” − “man” + “woman” ≈ “queen.” More recently, SAE experiments show that steering vectors — adding a feature direction to all positions in the residual stream — reliably induce the corresponding behavior. The linearity hypothesis is an empirical claim that has held in enough cases to be a useful working assumption, while having known limits.
Polysemanticity is the main obstacle to feature identification at the neuron level. When more features than neurons are needed, superposition allows multiple features to be stored in overlapping combinations — the network learns to use nearly-orthogonal directions in a high-dimensional space, packing more representational capacity per neuron at the cost of interference. SAEs extract features by training a dictionary of vectors and learning to represent activations as sparse combinations of dictionary vectors; sparsity prevents a single feature from spreading across many dictionary elements.
The SAE approach faces its own methodological questions: how to evaluate whether a learned SAE feature is genuinely monosemantic (corresponds to a single concept) rather than an artifact of the training procedure; how many features a model has in a given layer; whether features identified by automated methods match the features that drive interesting model behaviors. Activation steering — identifying a feature direction and artificially adding it to the residual stream — provides a causal test: a feature direction that causally induces the predicted behavior is more likely to be genuine than one that merely correlates with it.
Circuits: How Features Compose
A circuit is a subgraph of the computational graph that implements a specific function. Identifying a circuit requires: identifying the features at each layer, identifying which neurons or attention heads read and write those features, and verifying that the subgraph causally mediates the specific behavior being analyzed.
The induction head circuit is the clearest example. The circuit consists of two attention heads operating across two layers. Head A (the “previous token head”) attends to the token immediately preceding the current position. Head B (the “induction head”) attends to positions that followed the same token in earlier context. Together they implement: if token T appears again, predict what followed T before. This is provably sufficient for a wide range of in-context learning tasks, and activation patching experiments confirm that both heads are causally necessary for the behavior.
The key methodological tool is activation patching (also called causal tracing): run the model on two inputs that differ in some specific way, record the activations from both runs, then replace activations from one run with activations from the other at specific positions and layers. If replacing activations from the “clean” run restores the original behavior, those activations causally mediate the behavior difference. Meng et al.’s ROME paper used this to localize factual associations to specific MLP layers, finding that “The Eiffel Tower is in [Paris]” is stored in particular layers and can be edited by modifying the corresponding MLP weights.
The difficulty of circuit analysis is scale. Identifying a circuit manually requires forming hypotheses, running interventions, and iterating — a process that takes weeks or months for a single circuit in a small model. Production models have hundreds of layers and millions of attention heads. Automated circuit discovery (using methods like Edge Attribution Patching and flow-based attribution) attempts to identify circuits algorithmically, but current methods work on small models and specific behaviors, not at production scale.
What Can and Cannot Be Verified
Interpretability claims range from well-supported to speculative, and the difference matters. The key standard is causal verification: a claim about an internal mechanism is supported when intervening on that mechanism — activating or suppressing it, replacing its activations with those from another context — produces the predicted downstream effects.
Several failure modes reduce confidence in interpretability claims. Saliency methods (gradient-based importance scores) show which input features are most influential but can be highly sensitive to implementation details, inconsistent with perturbation tests, and systematically misleading about what the model actually uses. Attention visualization shows which positions a head attends to but does not establish that those positions are causally relevant — a head may attend to a position while transmitting information that was actually important through other channels. Post-hoc rationalizations are explanations that seem consistent with model behavior but are not uniquely implied by it — many different stories might be consistent with the same input-output behavior.
The practical implication is that claims in the interpretability literature require reading with calibration: what kind of evidence is provided? Is there causal intervention, or only correlation? Has the finding been replicated? Is the feature or circuit finding specific and falsifiable, or vague enough that it cannot be disproved? The literature contains work of varying rigor, and the same skepticism applied to ML benchmark results applies here.
What Studying This Changes
Mechanistic interpretability changes how practitioners relate to the models they use and build.
The first change is the transition from black-box testing to internal diagnosis. A practitioner without interpretability training can test a model extensively on representative inputs and draw inferences about its capabilities. A practitioner with interpretability training can ask structural questions about specific behaviors: what features are the model tracking in this layer? Which attention heads are responsible for this behavior? Is there a circuit that could fail in a way that testing has not covered? The diagnostic capability is qualitatively different from behavioral testing.
The second change is calibrated skepticism about model reliability. Polysemanticity means that neurons you cannot identify as specific-feature detectors may be involved in many behaviors in ways you haven’t characterized. Superposition means that adding a new feature to the training data may nonlinearly reorganize representations elsewhere. These structural properties of networks imply that behavioral testing, however extensive, leaves important unknowns — and the practitioner who understands these properties can identify which kinds of testing gaps are concerning and which can be closed by additional testing.
The third change is access to the safety-relevant questions. Whether a model contains internal representations consistent with its stated goals, whether fine-tuning on good behavior reliably changes internal representations or only surface patterns, whether a model that passes safety evaluations has concerning representations that are not expressed in current context — these questions require interpretability tools. Practitioners who understand the current state of the art know what can and cannot be investigated, and can make informed assessments of which deployment scenarios require mechanistic investigation rather than behavioral testing.
Resources
Books and Survey Articles
Molnar’s Interpretable Machine Learning (3rd ed., free at christophm.github.io/interpretable-ml-book) is the most comprehensive textbook on the broader interpretability field, covering feature importance, surrogate models, LIME, SHAP, attention visualization, and mechanistic methods. For learners who need the full landscape, it is the right starting point. For learners who specifically want the mechanistic tradition, the Distill articles are more directly relevant.
The Distill articles on Feature Visualization and Circuits (2017–2020, free at distill.pub) constitute the founding literature of mechanistic interpretability. Olah et al.’s “Feature Visualization” (2017), “The Building Blocks of Interpretability” (2018), and the Circuits thread (2020) should be read sequentially. The interactive visualizations convey what static descriptions cannot.
Elhage et al.’s “A Mathematical Framework for Transformer Circuits” (Anthropic, 2021, free) is the conceptual foundation for transformer interpretability. It is demanding — it presupposes fluency with transformer architecture at the implementation level — but for anyone doing mechanistic interpretability of language models, it is required reading.
For a self-contained introduction suitable for someone with ML foundations but without transformer expertise, Neel Nanda’s “Concrete Steps to Get Started in Transformer Mechanistic Interpretability” (free, available on his website) is the right entry.
| Resource | Role | Type |
|---|---|---|
| Molnar, Interpretable Machine Learning (3rd ed., free) | Broad interpretability landscape | Entry |
| Distill Feature Visualization + Circuits series (free) | Founding mechanistic literature | Practice |
| Elhage et al., “Mathematical Framework for Transformer Circuits” (2021, free) | Transformer interpretability foundation | Depth |
| Olsson et al., “In-context Learning and Induction Heads” (2022, free) | Foundational circuit discovery | Depth |
| Elhage et al., “Toy Models of Superposition” (2022, free) | Polysemanticity foundation | Depth |
| Bricken et al., “Towards Monosemanticity” (2023, free) | Sparse autoencoder foundation | Depth |
| Templeton et al., “Scaling Monosemanticity” (2024, free) | SAE at production scale | Depth |
| Meng et al., ROME paper (2022, free) | Factual association and patching | Depth |
| Nanda, “Concrete Steps for Mechanistic Interpretability” (free) | Accessible practical entry | Entry |
| Lipton, “The Mythos of Model Interpretability” (2018, free) | Essential skeptical counterpoint | Depth |
Courses and Lectures
There is no standard university course on mechanistic interpretability — the field is too new and too fast-moving. The best structured path is Nanda’s ARENA tutorials (Alignment Research Engineer Accelerator, free at arena.education), which provide a hands-on sequence: implementing transformers from scratch, performing activation analysis, replicating key interpretability findings. The tutorials are the closest thing to a curriculum in the field and are maintained by an active community.
Neel Nanda’s YouTube channel (free) provides lecture-style explanations of key papers and concepts, with particular strength on sparse autoencoders and transformer circuits.
AI Safety Fundamentals: AI Alignment (BlueDot Impact, free at aisafetyfundamentals.com) provides a structured reading sequence that includes mechanistic interpretability alongside the broader AI safety context that motivates it.
| Course | Platform | Type |
|---|---|---|
| ARENA mechanistic interpretability tutorials (free) | arena.education | Practice |
| Nanda YouTube lectures (free) | YouTube | Entry |
| AI Safety Fundamentals alignment course (free) | aisafetyfundamentals.com | Entry |
Practice, Tools, and Current Sources
TransformerLens (Neel Nanda, free, pip install transformer-lens) is the standard library for mechanistic interpretability of transformer models. It provides hooks into every attention head, MLP layer, and residual stream position, with utilities for activation patching, logit attribution, and circuit analysis. The library supports GPT-2, GPT-Neo, Pythia, and other open-weight models. Learning TransformerLens — working through the tutorial notebooks — is the practical entry to transformer interpretability.
SAELens (Joseph Bloom, free) provides tools for training and using sparse autoencoders, with pre-trained SAEs for GPT-2, Pythia, and other models. Working through the SAELens tutorials reveals what sparse features look like in practice and how steering experiments work.
The Neuroscope (Adam Jermyn and collaborators, free) provides a web interface for exploring individual neuron activations in GPT-2 — what text patterns maximally activate each neuron, with examples. Using it before studying feature visualization methods provides intuition about what you are trying to characterize formally.
Implementing the induction heads paper from scratch is the most instructive single project in the field: build a two-layer transformer in TransformerLens, identify which attention heads form induction heads, verify this by activation patching, and confirm that ablating the induction heads degrades in-context learning performance. The project takes a few days and produces the kind of hands-on intuition that reading cannot.
| Resource | Platform | Type |
|---|---|---|
| TransformerLens (free) | pip install transformer-lens | Practice |
| SAELens (free) | GitHub / pip | Practice |
| Neuroscope (free) | neuroscope.io | Practice |
| Neuronpedia (free) | neuronpedia.org | Practice |
| Gemma Scope (free) | deepmind.google | Reference |
| Anthropic Transformer Circuits Thread (free, ongoing) | transformer-circuits.pub | Reference |
| ARENA tutorial notebooks (free) | arena.education | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Equating saliency maps with mechanistic interpretability | Saliency methods (gradient-based attribution, LIME, SHAP, attention visualization) are interpretability tools that describe relationships between inputs and outputs. They do not characterize internal mechanisms, can be sensitive to implementation details, and have documented failure modes including inconsistency with perturbation tests and susceptibility to adversarial manipulations. Practitioners who learn saliency methods and consider themselves interpretability practitioners have learned the simpler and weaker tradition. | Study the documented failures of saliency methods (Adebayo et al.’s “Sanity Checks for Saliency Maps,” among others) before relying on them. Then study the mechanistic methods — activation patching, circuit analysis, SAEs — that provide causal verification. Use saliency as a starting point for generating hypotheses, not as evidence for mechanisms. |
| Trusting unverified interpretability claims | The interpretability literature contains many claims about what models “represent” or what circuits “implement” that are supported by correlation rather than causal intervention. A neuron that activates on the word “cat” might be detecting the concept “animal” more broadly, or tracking syntactic properties of certain words, or responding to something entirely different. Correlational findings are easier to produce and publish than causal ones. | Apply the causal verification standard: does intervening on this mechanism — through activation patching, ablation, or direct steering — produce the predicted effect? If not, the interpretability claim is not established. Work through Meng et al.’s ROME paper as a model of what causal verification looks like in practice. |
| Treating polysemanticity as a minor implementation detail | Polysemanticity — the fact that most neurons encode multiple unrelated features simultaneously through superposition — is not a quirk to be worked around. It is a fundamental property of how sufficiently expressive networks store information, and it means that neuron-level interpretation is systematically insufficient as a primary methodology. Much early interpretability work was done before polysemanticity was well-understood, and those findings have required revision. | Engage with the superposition literature (Elhage et al. 2022) before forming conclusions about what individual neurons represent. Use SAE-based feature discovery rather than neuron-level analysis as the primary methodology for feature characterization. Treat neuron activations as symptoms of underlying features rather than as features themselves. |
| Over-interpreting sparse autoencoder features | SAEs find features that are meaningful and causally effective, which is genuinely impressive. But the features they find are shaped by the training procedure (the loss, the dictionary size, the training data), and not all learned features correspond to concepts that are causally central to model behavior. SAE features should be verified through steering experiments, not just assumed to be mechanistically significant because they were found by the SAE. | For any SAE feature claimed to be mechanistically relevant, perform the steering experiment: artificially activate the feature direction and observe whether model behavior changes as predicted. Also perform the reverse: when the model is exhibiting the behavior, does the feature activate? Correlation in both directions strengthens the claim. |
| Skipping the hands-on implementation | Mechanistic interpretability is unusually dependent on implementation experience. The difference between reading about activation patching and doing activation patching is qualitative: the implementation reveals how residual streams actually work, how attention patterns look in practice, and what interpretability actually costs computationally. The ARENA tutorials exist specifically because the founders of the field recognized that reading was insufficient. | Work through the ARENA tutorials before engaging with the primary literature. Implement the induction heads finding from scratch. Only after getting your hands into TransformerLens and experiencing what activation patching produces will the primary papers be fully legible. |
| Treating the field’s results as complete | Mechanistic interpretability has found genuine results — induction heads, factual storage in MLP layers, SAE features — but it has also found more questions than answers. Whether current methods scale to frontier models is unknown. Whether the features that can be identified with current methods are the ones relevant for safety is unknown. Whether circuit-level analysis can characterize anything close to the full range of model behaviors is unknown. | Engage with the open questions explicitly. Read Lipton’s “Mythos of Model Interpretability” as a counterpoint to the optimistic framing. Ask, for each claimed interpretability finding, whether it scales to models that are actually deployed and whether the mechanism found is relevant to the safety properties that motivated the investigation. Maintain calibrated uncertainty about what the field has and has not achieved. |
中文
神经网络是一个函数。给定输入，它会产生输出。训练会调整它的参数，直到输出足够好地匹配目标。这个描述是完整的，但它几乎没有告诉你这个函数内部到底在做什么——它执行了哪些计算，追踪了哪些概念，中间层如何在从输入到输出的过程中处理信息。可解释性就是回答这些问题的学科：它不只问网络输出了什么，而是问它正在做什么。
更广义的可解释性文献包括注意力可视化和显著性图（saliency maps）等技术，它们会突出输入中哪些部分对输出影响最大。这些方法有用，但有限：它们描述的是输入与输出之间的关系，而不是刻画内部计算。机制可解释性走得更深，它试图逆向工程网络内部真正的计算回路——识别单个神经元检测什么特征，这些特征组成什么回路来计算特定函数，以及这些回路如何组合成模型的整体行为。目标是得到能经受因果检验的理解，而不是那些看起来像解释、却未必反映模型实际运行方式的模式。
这门学科与 AI 安全有直接而结构性的关系。一个内部机制被理解的模型，是一个可以在新条件、对抗性输入或能力扩展下推理其行为的模型。一个黑箱模型只能被测试；它不能被保证。随着语言模型被部署到医疗、法律、安全等高风险场景中，“被广泛测试过”和“机制被理解”之间的差距，会变成一个具有现实后果的工程差距。
前置知识：深度学习（§5.3）——Transformer 架构细节、注意力机制、残差流。机器学习基础（§5.2）——线性代数，达到能理解激活如何作为特征线性组合的水平。大语言模型（§5.5）——当代可解释性最主要研究的模型对象。
从特征可视化到机制理解
作为一门系统性的经验学科，可解释性研究的历史出人意料地短。它所需要的材料——具有有趣内部结构的已训练神经网络——从早期深度学习时代就已经存在。但系统性理解这些结构的项目，主要是过去十年的产物；大多数核心概念词汇是在 2017 到 2023 年间建立起来的。
奠基性工作是一系列由 Chris Olah 及其同事完成的论文，最终集中体现在他 2017 年发表于 Distill 的文章 “Feature Visualization” 中。核心观察很简单：你可以通过寻找最能激活某个神经元的输入，来理解这个神经元检测什么——也就是对输入进行梯度上升，使该神经元的激活最大化。这为卷积网络中的神经元产生了可解释图像：早期层的神经元会被边缘和 Gabor 模式最大激活；后期层的神经元则会被人脸、动物或特定纹理最大激活。这个技术本身并不新——DeepDream 在 2015 年已经使用过类似的梯度上升——但系统、细致地应用它来理解单个神经元到底在计算什么，则是新的。Distill 的形式配合交互式可视化，也让这些发现变得异常易懂。
Circuits 系列文章（Olah、Cammarata、Schubert 等，2020）把特征可视化从单个神经元扩展到神经元之间的连接。关键洞见是：神经元并不是孤立的；它们会形成回路：由权重连接起来的特征组合，实现特定计算功能。论文识别了卷积网络中的具体回路——曲线检测器、高低频检测器、多频检测器——并一步步追踪它们执行的计算。这种方法明确受神经科学启发：“circuit” 借用了神经科学中局部回路在更大脑区内执行特定功能的词汇。这种类比是否在深层意义上适用于生物回路仍有争议，但识别功能单元并刻画其计算的做法，已经证明很有生产力。
从卷积网络转向 Transformer，需要新的方法。Elhage 等人的 “A Mathematical Framework for Transformer Circuits”（Anthropic，2021）建立了 Transformer 可解释性的概念词汇：残差流是通信通道，每一层都会从中读取并向其中写入；注意力头是作用在残差流上的低秩操作；MLP 层是 key-value 记忆。论文描述了一层和两层 Transformer 可以执行的具体计算，为思考 Transformer 正在计算什么，而不只是输出什么，提供了第一个系统性框架。
induction heads 论文（Olsson、Elhage、Nanda 等，2022）首次在生产规模语言模型中发现了一个具体回路。Induction heads 是一个由两个注意力头组成的回路，它使上下文学习成为可能：一个注意力头把此前出现过的 token 模式复制到当前上下文，第二个注意力头则关注此前跟在相同 token 之后的位置。这个回路实现了上下文学习背后的基本机制——如果你曾经见过 “[A] [B] … [A]”，那就预测 “[B]”——论文还证明，这个回路会在训练过程中形成，会出现在不同架构中，并且会因果性地中介模型完成模式补全的能力。因果验证——证明破坏该回路会以精确预测的方式降低性能——是一项方法论进步：它不只是说“这个回路存在”，而是说“这个回路正在做这件事”。
多义性问题（polysemanticity）打断了简单的特征分析。一个干净的故事会是：单个神经元表示单个特征——一个神经元表示“微笑”，一个表示“狗”，一个表示“过去时”。但逐渐积累的事实是，大多数神经元都是多义的：它们会同时响应多个彼此无关的特征。这不是噪声；它反映了网络如何存储比神经元数量更多的特征这一数学事实：网络利用高维几何，使大量特征即使多于神经元，也能保持近似正交。Elhage 等人的 “Toy Models of Superposition”（2022）给出了第一个系统性的理论和经验处理：它展示了叠加何时发生、为什么有利，以及由此产生的表示几何是什么样子。
多义性使神经元层面的分析出现根本限制。如果每个神经元都编码多个特征，那么理解一个神经元并不等于得到了特征。回应方法是稀疏自编码器（sparse autoencoders，SAEs）：训练一个辅助网络，以激活作为输入，并把它分解为若干学习得到的方向的稀疏组合，其中每个方向对应一个可解释的单一特征。Bricken 等人的 “Towards Monosemanticity”（2023）证明，在一个玩具设定下，对单层 Transformer 训练 SAE 可以产生数千个可解释的、单义的特征。Templeton 等人的 “Scaling Monosemanticity”（2024）把这一方法扩展到 Claude 3 Sonnet，发现了数百万个可解释特征，包括抽象概念：“Golden Gate Bridge”“banana”“Martin Luther King”，以及“迎合性概念”，都可以被识别为激活空间中的连贯方向。人为激活“banana”特征，会导致模型在没有提示的情况下谈论香蕉；激活“sycophancy”特征，会改变模型在同意用户和保持准确之间的平衡。这些都是因果展示——特征在被直接刺激时，会机械性地影响模型行为。
当前前沿在于研究：特征是否可以组合成更高层次的回路，回路是否能被自动大规模识别（回路发现目前很大程度上仍是手工且劳动密集的），以及可解释性方法是否能够揭示与安全相关的内部结构：一个表面上看起来有帮助的模型，其内部计算是否真的与此一致，还是某些令人担忧的表示可能隐藏在表面行为之下。可解释性与安全的交叉，是多数应用动机的来源。
特征、回路，以及可知性的边界
模型表示了什么特征
特征是机制可解释性的基本单位。一个特征是激活空间中的一个方向——也就是神经元激活的线性组合——并且对应一个人类可解释的概念。关键结果是：特征是线性的。一个模型是否正在“思考”某个特定概念，对应于其激活空间中相应方向的分量权重是否较高。
线性证据来自多个方向。词嵌入著名地展示了线性几何：“king” − “man” + “woman” ≈ “queen”。更近期的 SAE 实验表明，steering vectors——也就是把一个特征方向加到残差流的所有位置上——可以可靠地诱导出相应行为。线性假设是一个经验主张；它已经在足够多案例中成立，因此可以作为有用的工作假设，同时它也有已知边界。
多义性是神经元层面特征识别的主要障碍。当所需特征多于神经元数量时，叠加允许多个特征被存储在重叠组合中——网络学会在高维空间中使用近似正交方向，从而以干扰为代价，让每个神经元承载更多表征容量。SAE 通过训练一个向量字典，并学习把激活表示为字典向量的稀疏组合来提取特征；稀疏性可以防止单个特征扩散到许多字典元素中。
SAE 方法本身也面临方法论问题：如何评估一个学到的 SAE 特征确实是单义的，也就是对应单一概念，而不是训练过程的产物？一个模型在某一层中到底有多少特征？自动方法识别出的特征，是否匹配那些真正驱动有趣模型行为的特征？激活引导（activation steering）提供了一种因果检验：识别一个特征方向，并人为把它加入残差流。如果这个特征方向能因果性地诱导出预测行为，那么它比仅仅与行为相关的方向更可能是真实特征。
回路：特征如何组合
回路是计算图中实现特定函数的子图。识别一个回路需要：识别每一层的特征，识别哪些神经元或注意力头读取和写入这些特征，并验证该子图是否因果性地中介被分析的具体行为。
induction head 回路是最清晰的例子。这个回路由跨越两层的两个注意力头组成。Head A（“previous token head”）关注当前位置之前的那个 token。Head B（“induction head”）关注早期上下文中跟在相同 token 之后的位置。它们共同实现了：如果 token T 再次出现，就预测此前跟在 T 后面的内容。对于广泛的上下文学习任务，这一机制已经足够；激活 patching 实验证实，这两个头对该行为都是因果必要的。
关键方法论工具是激活 patching（也称 causal tracing）：在两个以特定方式不同的输入上运行模型，记录两次运行的激活，然后在特定位置和层上，用一次运行的激活替换另一次运行的激活。如果用“干净”运行中的激活替换后恢复了原始行为，那么这些激活就因果性地中介了行为差异。Meng 等人的 ROME 论文用这种方法把事实关联定位到特定 MLP 层，发现 “The Eiffel Tower is in [Paris]” 这样的事实存储在特定层中，并且可以通过修改相应 MLP 权重来编辑。
回路分析的困难在于规模。手动识别一个回路，需要形成假设、运行干预、反复迭代——即使是在小模型中分析单个回路，也可能需要数周或数月。生产模型有数百层和数百万个注意力头。自动回路发现（使用 Edge Attribution Patching 和基于流的归因等方法）试图用算法识别回路，但当前方法适用于小模型和特定行为，还无法在生产规模上工作。
什么可以被验证，什么不能
可解释性主张从证据充分到相当推测不等，区分二者很重要。关键标准是因果验证：当对内部机制进行干预——激活或抑制它，或用另一个语境中的激活替换它——能够产生预测的下游效果时，关于内部机制的主张才算得到支持。
几种失败模式会降低对可解释性主张的信心。显著性方法（基于梯度的重要性分数）会显示哪些输入特征最有影响，但它们可能对实现细节高度敏感，与扰动测试不一致，并且会系统性误导我们理解模型实际使用了什么。注意力可视化会显示一个注意力头关注哪些位置，但这并不能证明这些位置具有因果相关性——一个头可能关注某个位置，但真正重要的信息通过其他通道传输。事后合理化解释是那些看起来与模型行为一致、但并不是由模型行为唯一推出的解释——许多不同故事都可能与同一个输入—输出行为相容。
实际含义是：阅读可解释性文献中的主张时，需要保持校准。它提供了哪种证据？是因果干预，还是只有相关性？发现是否被复现？特征或回路发现是否具体、可证伪，还是模糊到无法被推翻？这个文献包含严谨程度不同的工作；我们对 ML 基准结果所采取的怀疑，也应当用于这里。
学习这一部分会改变什么
机制可解释性会改变实践者与他们使用和构建的模型之间的关系。
第一个变化，是从黑箱测试转向内部诊断。没有可解释性训练的实践者，可以在代表性输入上广泛测试模型，并据此推断其能力。具备可解释性训练的实践者，则可以对具体行为提出结构性问题：模型在这一层追踪了哪些特征？哪些注意力头负责这个行为？是否存在一个回路可能以测试没有覆盖到的方式失败？这种诊断能力在性质上不同于行为测试。
第二个变化，是对模型可靠性形成校准过的怀疑。多义性意味着，那些你无法识别为特定特征检测器的神经元，可能以尚未刻画的方式参与许多行为。叠加意味着，把一个新特征加入训练数据，可能会非线性地重组其他地方的表示。这些网络结构属性意味着，无论行为测试多么广泛，仍然会留下重要未知；理解这些属性的实践者，能够识别哪些测试缺口令人担忧，哪些可以通过额外测试关闭。
第三个变化，是能够进入与安全相关的问题。一个模型内部是否包含与其声称目标一致的表征？在良好行为上微调是否可靠地改变了内部表征，还是只改变了表面模式？一个通过安全评估的模型，是否含有在当前语境中没有表达出来的令人担忧表征？这些问题需要可解释性工具。理解当前技术水平的实践者知道什么可以被调查、什么还不能，并能判断哪些部署场景需要机制层面的调查，而不只是行为测试。
资源
书籍与综述文章
Molnar 的 Interpretable Machine Learning（第 3 版，可在 christophm.github.io/interpretable-ml-book 免费获取）是更广义可解释性领域中最全面的教材，覆盖特征重要性、替代模型、LIME、SHAP、注意力可视化和机制方法。对于需要完整图景的学习者，这是合适起点。对于专门想学习机制传统的学习者，Distill 文章更直接相关。
Distill 上关于 Feature Visualization 和 Circuits 的文章（2017–2020，可在 distill.pub 免费获取）构成了机制可解释性的奠基文献。Olah 等人的 “Feature Visualization”（2017）、“The Building Blocks of Interpretability”（2018）以及 Circuits 系列（2020）应当按顺序阅读。交互式可视化能传达静态描述无法传达的内容。
Elhage 等人的 “A Mathematical Framework for Transformer Circuits”（Anthropic，2021，免费）是 Transformer 可解释性的概念基础。它要求很高——预设读者在实现层面对 Transformer 架构很熟悉——但对于任何做语言模型机制可解释性的人来说，它都是必读材料。
对于有 ML 基础但没有 Transformer 专业知识的读者，Neel Nanda 的 “Concrete Steps to Get Started in Transformer Mechanistic Interpretability”（免费，可在他的网站上获取）是合适入口。
| 资源 | 作用 | 类型 |
|---|---|---|
| Molnar, Interpretable Machine Learning（第 3 版，免费） | 广义可解释性图景 | 入门 |
| Distill Feature Visualization + Circuits series（免费） | 机制可解释性奠基文献 | 实践 |
| Elhage et al., “Mathematical Framework for Transformer Circuits”（2021，免费） | Transformer 可解释性基础 | 深入 |
| Olsson et al., “In-context Learning and Induction Heads”（2022，免费） | 基础回路发现 | 深入 |
| Elhage et al., “Toy Models of Superposition”（2022，免费） | 多义性基础 | 深入 |
| Bricken et al., “Towards Monosemanticity”（2023，免费） | 稀疏自编码器基础 | 深入 |
| Templeton et al., “Scaling Monosemanticity”（2024，免费） | 生产规模 SAE | 深入 |
| Meng et al., ROME paper（2022，免费） | 事实关联与 patching | 深入 |
| Nanda, “Concrete Steps for Mechanistic Interpretability”（免费） | 易上手的实践入口 | 入门 |
| Lipton, “The Mythos of Model Interpretability”（2018，免费） | 必要的怀疑性反论 | 深入 |
课程与讲座
机制可解释性目前还没有标准大学课程——这个领域太新，变化也太快。最好的结构化路径是 Nanda 的 ARENA tutorials（Alignment Research Engineer Accelerator，可在 arena.education 免费获取），它提供一套动手序列：从零实现 Transformer，进行激活分析，复现关键可解释性发现。这些教程是目前最接近课程体系的材料，并由一个活跃社区维护。
Neel Nanda 的 YouTube 频道（免费）提供关键论文和概念的讲座式解释，尤其擅长讲稀疏自编码器和 Transformer 回路。
AI Safety Fundamentals: AI Alignment（BlueDot Impact，可在 aisafetyfundamentals.com 免费获取）提供结构化阅读序列，把机制可解释性放在更广泛的 AI 安全语境中。
| 课程 | 平台 | 类型 |
|---|---|---|
| ARENA mechanistic interpretability tutorials（免费） | arena.education | 实践 |
| Nanda YouTube lectures（免费） | YouTube | 入门 |
| AI Safety Fundamentals alignment course（免费） | aisafetyfundamentals.com | 入门 |
实践、工具与当前资料
TransformerLens（Neel Nanda，免费，pip install transformer-lens）是 Transformer 模型机制可解释性的标准库。它可以 hook 到每个注意力头、MLP 层和残差流位置，并提供激活 patching、logit attribution 和回路分析工具。该库支持 GPT-2、GPT-Neo、Pythia 和其他开放权重模型。学习 TransformerLens——完成教程 notebook——是进入 Transformer 可解释性实践的入口。
SAELens（Joseph Bloom，免费）提供训练和使用稀疏自编码器的工具，并包含 GPT-2、Pythia 和其他模型的预训练 SAE。完成 SAELens 教程，可以看到稀疏特征在实践中是什么样子，以及 steering 实验如何工作。
Neuroscope（Adam Jermyn 及合作者，免费）提供了一个网页界面，用来探索 GPT-2 中单个神经元的激活——哪些文本模式会最大激活每个神经元，并给出示例。在学习特征可视化方法之前使用它，可以帮助建立直觉，理解你真正想要形式化刻画的是什么。
从零实现 induction heads 论文，是这个领域最有教学价值的单个项目：在 TransformerLens 中构建一个两层 Transformer，识别哪些注意力头形成 induction heads，通过激活 patching 验证这一点，并确认消融 induction heads 会降低上下文学习表现。这个项目需要几天时间，但会产生单靠阅读无法获得的动手直觉。
| 资源 | 平台 | 类型 |
|---|---|---|
| TransformerLens（免费） | pip install transformer-lens |
实践 |
| SAELens（免费） | GitHub / pip | 实践 |
| Neuroscope（免费） | neuroscope.io | 实践 |
| Neuronpedia（免费） | neuronpedia.org | 实践 |
| Gemma Scope（免费） | deepmind.google | 参考 |
| Anthropic Transformer Circuits Thread（免费，持续更新） | transformer-circuits.pub | 参考 |
| ARENA tutorial notebooks（免费） | arena.education | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 把显著性图等同于机制可解释性 | 显著性方法（基于梯度的归因、LIME、SHAP、注意力可视化）是描述输入和输出之间关系的可解释性工具。它们并不刻画内部机制，可能对实现细节敏感，也有已记录的失败模式，包括与扰动测试不一致、容易受到对抗性操纵。学习了显著性方法就以为自己是可解释性实践者，只是学到了更简单也更弱的传统。 | 在依赖显著性方法之前，先学习其已记录的失败案例，例如 Adebayo 等人的 “Sanity Checks for Saliency Maps”。然后再学习提供因果验证的机制方法——激活 patching、回路分析、SAE。把显著性作为产生假设的起点，而不是作为机制证据。 |
| 信任未验证的可解释性主张 | 可解释性文献中有许多关于模型“表示”了什么、回路“实现”了什么的主张，但它们只是由相关性支持，而不是由因果干预支持。一个在 “cat” 这个词上激活的神经元，可能检测的是更广义的“动物”概念，也可能追踪某些词的句法属性，甚至可能在响应完全不同的东西。相关性发现比因果发现更容易产生，也更容易发表。 | 使用因果验证标准：对这个机制进行干预——通过激活 patching、消融或直接 steering——是否会产生预测效果？如果不会，这个可解释性主张就没有成立。阅读 Meng 等人的 ROME 论文，把它作为实践中因果验证是什么样子的范例。 |
| 把多义性当成小的实现细节 | 多义性——大多数神经元通过叠加同时编码多个彼此无关的特征——并不是一个可以绕过去的小怪癖。它是足够有表达力的网络存储信息的根本性质，意味着神经元层面的解释作为主要方法是系统性不足的。许多早期可解释性工作是在多义性尚未被充分理解前完成的，这些发现后来需要修正。 | 在对单个神经元表示什么形成结论之前，先学习叠加文献（Elhage 等，2022）。使用基于 SAE 的特征发现，而不是把神经元层面分析作为主要特征刻画方法。把神经元激活视为底层特征的症状，而不是特征本身。 |
| 过度解释稀疏自编码器特征 | SAE 找到的特征有意义并且具有因果效果，这确实令人印象深刻。但它们找到的特征会受训练过程影响，包括损失函数、字典大小和训练数据；并非所有学到的特征都对应那些对模型行为具有因果中心性的概念。SAE 特征应当通过 steering 实验验证，而不能仅仅因为是 SAE 找到的，就被假定为机制上重要。 | 对任何被声称具有机制相关性的 SAE 特征，都执行 steering 实验：人为激活该特征方向，观察模型行为是否按预测变化。也要做反向检查：当模型表现出该行为时，这个特征是否激活？两个方向的相关性都会增强该主张。 |
| 跳过动手实现 | 机制可解释性格外依赖实现经验。阅读激活 patching 和真正做激活 patching 之间，是质的差异：实现会揭示残差流实际如何工作、注意力模式在实践中长什么样，以及可解释性到底需要多少计算成本。ARENA 教程之所以存在，正是因为这个领域的奠基者意识到，只阅读是不够的。 | 在进入一手文献之前，完成 ARENA 教程。从零实现 induction heads 发现。只有真正动手使用 TransformerLens，并体验激活 patching 会产生什么，一手论文才会变得完全可读。 |
| 把这个领域的结果看成已经完整 | 机制可解释性已经发现了一些真实结果——induction heads、MLP 层中的事实存储、SAE 特征——但它也发现了更多问题。当前方法能否扩展到前沿模型，仍然未知。当前方法能识别出的特征，是否正是与安全相关的特征，仍然未知。回路层面的分析能否刻画接近完整范围的模型行为，也仍然未知。 | 明确面对开放问题。把 Lipton 的 “Mythos of Model Interpretability” 作为对乐观叙述的反论来阅读。对于每个被声称的可解释性发现，都要追问：它是否能扩展到实际部署的模型？它发现的机制是否与最初激发研究的安全属性有关？对这个领域已经做到什么、尚未做到什么，保持校准过的不确定性。 |