English
A deep learning system is a composition of many layers of differentiable functions whose parameters are jointly trained to minimize a loss on data. The depth is the key: a single layer of a neural network can compute any function given sufficient width, but an exponentially large network may be required to do so. Multiple layers learn a hierarchy of representations — early layers detecting simple patterns, later layers composing them into increasingly abstract ones — in a way that encodes useful inductive biases for data with hierarchical structure. For images, that structure is spatial; for language, it is sequential and compositional; for molecular biology, it is geometric. Choosing the right architecture encodes the right inductive bias for the structure present in the data.
The training procedure is the second defining element. Backpropagation — applying the chain rule to compute gradients through the composition of functions — makes it possible to train networks with many layers using gradient descent. This computation is automatic: given a differentiable computational graph and a loss function, automatic differentiation computes the gradient of the loss with respect to every parameter in one backward pass. The combination of expressive architectures, automatic differentiation, and large datasets has produced systems that perform at or above human level on perception, generation, and reasoning tasks that were considered intractable for decades.
This section builds directly on §5.2. The statistical learning theory, optimization concepts, and evaluation discipline from §5.2 apply throughout. What this section adds is the architectural vocabulary, training techniques, and the empirical phenomena specific to deep networks — particularly the puzzles around generalization in overparameterized models, emergent capabilities at scale, and the design of architectures that encode appropriate inductive biases.
Prerequisites: Machine learning foundations (§5.2) — everything there is assumed. Linear algebra (§2.3) at the level of matrix calculus. Calculus (§2.4) for gradients and the chain rule. Probability (§2.5) for understanding loss functions and generative models.
From Perceptrons to Transformers: The Empirical Development of a Discipline
The story of deep learning is the story of a good idea repeatedly rediscovered, extended past its theoretical support, and eventually vindicated by hardware, data, and scale.
Frank Rosenblatt’s perceptron (1957) learned to classify inputs by adjusting weights in response to errors — the first machine that genuinely learned a decision rule from examples with a convergence guarantee. The optimism it inspired was cut short in 1969 when Minsky and Papert’s Perceptrons proved that single-layer networks could not learn XOR, and — more importantly — appeared to argue that adding layers would not help. Neural network research contracted sharply. The first winter was not quite as absolute as legend has it: Paul Werbos derived backpropagation in his 1974 thesis, and Linnainmaa had computed the reverse-mode accumulation of derivatives in 1970. But the results were not connected to the right community at the right moment, and deep networks went largely unstudied for a decade.
The reconnection arrived in 1986. Rumelhart, Hinton, and Williams published “Learning Representations by Back-propagating Errors” in Nature, demonstrating that multi-layer networks trained by backpropagation could solve problems single-layer networks could not, and could develop internal representations that captured structure in the data. The paper’s influence was immediate and enormous: it reframed what neural networks were doing from “learning weights” to “learning representations,” and it made multi-layer networks the central object of study.
Yann LeCun took the next step by asking what representation structure a network should be designed to discover before training begins. For images, the answer was local spatial structure and translation invariance: patterns like edges and textures appear at various locations and should be detected wherever they appear. Convolutional neural networks (CNNs) — with shared weight filters applied at every spatial location — encoded this prior. LeCun’s LeNet (1989) read handwritten digits at the Federal Reserve using CNNs, and the system was so reliable that it processed about 10% of all checks written in the US by the late 1990s. This was deep learning’s first industrial-scale deployment, though it was not recognized as such at the time.
The 1990s saw the field fragment. SVMs had cleaner theory and strong performance on many benchmarks; neural networks required difficult-to-tune training and exhibited the vanishing gradient problem in deeper architectures. Hochreiter and Schmidhuber’s LSTM (1997) addressed the vanishing gradient for sequential data by introducing gates that controlled information flow across time, enabling recurrent networks to learn long-range dependencies. LeCun demonstrated CNNs continued working. But the general case of deep networks for arbitrary problems remained elusive, and the field’s center of gravity moved toward kernel methods.
The second wave was driven not by algorithmic innovation but by data and hardware arriving at a sufficient scale. The ImageNet dataset — 1.2 million labeled images across 1,000 categories, assembled by Fei-Fei Li and colleagues starting in 2009 — provided a benchmark of the right scale: large enough that hand-engineered features hit a ceiling, large enough that a deep network could learn to distinguish fine-grained categories. The GPU provided the computational capacity: training a deep convolutional network on ImageNet required weeks on CPUs but days on GPUs.
When Krizhevsky, Sutskever, and Hinton entered AlexNet in the 2012 ImageNet competition, it achieved a top-5 error of 15.3% — against the previous record of 26.2% and other 2012 entries in the 26-30% range. The gap was large enough to be unambiguous: deep learning had qualitatively changed what was possible in computer vision. Within two years, every top ImageNet entry was a deep convolutional network. The event restructured how the AI community understood what deep learning was capable of.
The years 2012–2017 were a period of rapid architectural development. Graves et al. trained LSTMs with CTC loss for speech recognition, achieving then-state-of-the-art results. Word2Vec (Mikolov et al., 2013) showed that skip-gram training on text produced word embeddings with semantic arithmetic properties — “king - man + woman ≈ queen” — demonstrating that distributed representations of language were genuinely learnable. ResNet (He et al., 2015) solved the degradation problem for very deep networks by introducing skip connections that allowed gradients to flow directly through identity mappings, enabling networks of 100+ layers. Batch normalization (Ioffe and Szegedy, 2015) made training faster and more stable by normalizing activations within each layer during training. GANs (Goodfellow et al., 2014) introduced the adversarial training framework for generative models. Variational autoencoders (Kingma and Welling, 2013) provided a probabilistic framework for representation learning and generation. Dropout regularization, deep networks for reinforcement learning (Atari, 2013), and sequence-to-sequence architectures all appeared within a few years.
The transformer, introduced by Vaswani et al. in “Attention Is All You Need” (2017), changed the structure of the field. Self-attention — each position in a sequence attending to all others — allowed models to capture long-range dependencies without the sequential computation that made LSTMs hard to parallelize. The transformer was faster to train at scale, achieved better results on translation, and generalized to tasks far outside machine translation. BERT (Devlin et al., 2018) showed that pre-training a large transformer on masked language modeling and then fine-tuning on downstream tasks dramatically outperformed task-specific architectures. GPT (Radford et al., 2018) showed that autoregressive language modeling on large text corpora produced representations useful for many tasks.
The discovery of scaling laws gave the current era its organizing principle. Kaplan et al.’s 2020 paper “Scaling Laws for Neural Language Models” showed that language model performance on held-out data improved as a smooth power law in compute, model size, and dataset size — without any indication of saturation. This was not merely a description of past results; it was a quantitative prediction about future results. If the power law held, larger models trained on more data with more compute would predictably perform better. The insight restructured the competitive landscape: the bottleneck shifted from algorithmic innovation to compute and data acquisition. The subsequent history — GPT-3, PaLM, Chinchilla, LLaMA, and the current generation of frontier models — is largely the story of organizations with different compute and data advantages following these scaling predictions.
What emerged from scale that was not present at smaller scales has been one of the field’s more surprising findings. In-context learning — the ability to solve tasks given only a few examples in the context window, without gradient updates — appears to emerge somewhere around 10–100 billion parameters and was not designed into any architecture. Chain-of-thought reasoning — where breaking a problem into explicit intermediate steps substantially improves accuracy — emerges similarly at scale. These emergent capabilities are not predicted by the behavior of smaller models in any obvious way, and their mechanism is only partially understood. They are also potentially unstable: tasks where models show emergent capability at one scale may show sudden drops when evaluation is made more rigorous.
The current deep learning landscape is heterogeneous. Transformers remain dominant for sequence tasks and have been extended to vision (Vision Transformers, ViT), audio, protein structure, and multi-modal settings. State-space models (Mamba) offer efficiency advantages for long sequences. Diffusion models dominate image, video, and audio generation. Mixture-of-experts architectures scale model capacity without proportionally increasing inference cost. What unites all of these is the same core training loop: differentiable architecture, automatic differentiation, stochastic gradient descent on very large datasets.
Architectures, Training, and What Scale Changes
Architecture as Inductive Bias
Every deep learning architecture encodes assumptions about what structure the data has, and the mismatch or fit between architectural assumptions and data structure determines whether the network learns effectively from a given amount of data.
Convolutional networks assume spatial locality and translation invariance: a filter applied at position (i,j) is the same filter applied at (i+10, j+10). This is an excellent assumption for natural images and audio spectrograms, where patterns are local and their meaning is position-invariant. It is a poor assumption for tabular data where features at positions 1 and 1001 may be highly related. The benefit is substantial: the number of parameters scales with filter size rather than input size, and the inductive bias dramatically reduces the amount of data required for learning.
Recurrent networks assume sequential structure: each step depends on all previous steps through a hidden state. LSTMs and GRUs add gating to control what information persists across steps. This assumption fits language, speech, and time-series data, but recurrent architectures are inherently sequential — each step requires the previous step’s output — making them difficult to parallelize.
Transformers assume that any position in a sequence can be related to any other position with equal cost, and the model should learn which relationships matter through attention weights. The self-attention mechanism computes queries, keys, and values for each position; the attention weight between positions i and j is proportional to exp(qᵢ · kⱼ / √d). This all-to-all connectivity is expressive and parallelizable but expensive: attention has O(n²) complexity in sequence length. Efficient attention variants (linear attention, sparse attention, flash attention for memory efficiency) are active engineering research.
Graph neural networks assume that the relevant structure is a graph with nodes and edges, and that information propagates by nodes aggregating information from their neighbors. This assumption fits molecular property prediction, social network analysis, and knowledge graph reasoning.
The choice of architecture is therefore the primary mechanism for incorporating domain knowledge before training. Architecture design has moved from heuristic craft to something approaching engineering: architectural properties like receptive field size, expressivity, and inductive bias can be analyzed, and the analysis guides design.
Training at Depth: Gradient Flow and Normalization
Training a deep network requires that gradients flow usably through many layers from the loss backward. Early deep networks suffered from the vanishing gradient problem: gradients shrank exponentially with depth, making parameters in early layers update negligibly. Several solutions have become standard.
Skip connections (residual connections) allow gradient to flow directly through identity mappings, bypassing the nonlinear transformation at each layer. ResNet demonstrated that with skip connections, networks of 100–1000 layers could be trained stably, dramatically extending the effective depth achievable.
Normalization layers — batch normalization, layer normalization, RMS normalization — stabilize training by normalizing the distribution of activations, reducing the internal covariate shift that destabilizes deep network training. Batch normalization normalizes within a batch; layer normalization normalizes within a single example (better for variable-length sequences). Different architectures use different normalization choices for reasons that are partially empirical.
Initialization matters: the variance of parameter initializations should be chosen to prevent gradients from exploding or vanishing at initialization. He initialization (for ReLU activations) and Xavier/Glorot initialization (for sigmoid and tanh) scale initial parameter variances based on layer width. Poor initialization can prevent training from succeeding even when the architecture and data would otherwise support it.
The optimizer interacts with the architecture. Adam’s adaptive per-parameter learning rates are particularly effective for transformers where different parameters have very different gradient magnitudes. Weight decay (L2 regularization) in the optimizer (AdamW vs. Adam with L2 loss regularization) has measurable effects on generalization due to the difference in how it interacts with the adaptive learning rates.
What Scale Changes: Emergent Capabilities and Scaling Laws
Scaling laws are empirical power-law relationships between performance and compute, model size, or data. For language models, Kaplan et al. found that cross-entropy loss on held-out text decreases as L ∝ N^{-α} where N is model size, with the exponent α around 0.07 for compute-optimal training. Hoffmann et al. (Chinchilla, 2022) showed that the Kaplan laws had an error: given a fixed compute budget, the optimal strategy is to train a smaller model on more data rather than a larger model on less data, revising the optimal model/data ratio.
Scaling laws are useful as planning tools: they predict, before training, approximately how much performance improvement to expect from a given investment in compute. They also reveal that there is no threshold or qualitative transition in the smooth part of the curve — improvement continues as a power law without saturation.
What scaling laws do not capture is emergent capabilities: behaviors that appear at larger scales and are not predicted by interpolating from smaller models. In-context learning, multi-step arithmetic, chain-of-thought reasoning, and the ability to follow complex instructions all appear to emerge above certain scale thresholds. These transitions are step-functions in the relationship between scale and capability — not smooth power laws — and they are incompletely understood. Wei et al.’s “Emergent Abilities of Large Language Models” (2022) documented these transitions; subsequent work has questioned whether some emergent abilities are artifacts of evaluation metrics or genuine capability transitions.
The distinction matters practically. If a capability is smoothly scaling, more compute reliably helps. If a capability emerges discontinuously, small-scale experiments provide no information about whether the capability will appear at larger scale. This makes empirical research on small models partially uninformative for predicting frontier model behavior, a structural challenge for the field.
What Studying This Changes
Deep learning changes how practitioners understand computational systems that process unstructured data.
The first change is architectural intuition: the ability to match data structure to architectural assumptions. Given a new learning problem, the question is not “which architecture is best” but “what structure does this data have, and which architecture’s inductive biases align with that structure?” This question has answers — spatial locality for images, sequential dependencies for language, graph structure for molecules — and the practitioner who has internalized the architectural vocabulary can engage with them.
The second change is training fluency. Deep networks can fail to train in many ways — gradient explosion, mode collapse, representation collapse, training-test gap from distribution shift. These failure modes have characteristic signatures in training curves and evaluation metrics, and the practitioner who has trained many networks develops diagnostic pattern recognition: “loss is oscillating, suggests learning rate too high” or “train loss is decreasing but val loss is flat, suggests overfitting from insufficient regularization.” This diagnostic capability is acquired through practice, but its foundation is understanding what each component of the training procedure is doing.
The third change is the ability to read the research literature productively. Deep learning papers have a characteristic structure — problem formulation, architecture description, experimental results, ablation studies — and reading them requires the vocabulary to evaluate whether the architectural choices are motivated, whether the baselines are appropriate, whether the ablations establish causality. The practitioner who can read a new paper and assess whether it represents genuine progress or a benchmarking artifact is operating at a different level from the practitioner who takes results at face value.
The fourth change is calibrated uncertainty about scale. The practitioner who has studied scaling laws and emergent capabilities knows when to trust small-scale experiments and when not to: smooth scaling behaviors are well-predicted by small experiments; emergent capabilities are not. This calibration prevents both over-reliance on small-scale results that may not hold at scale, and excessive skepticism about small-scale experiments that do predict large-scale behavior.
Resources
Books and Texts
Bishop and Bishop’s Deep Learning: Foundations and Concepts (2024) is the most current foundational text, written with mathematical depth and covering contemporary architectures including transformers and diffusion models. For learners beginning serious deep learning study now, this is the right primary reference.
Goodfellow, Bengio, and Courville’s Deep Learning (2016, free at deeplearningbook.org) remains valuable for its rigorous treatment of the mathematical foundations — linear algebra, probability, numerical computation, information theory — and for the pre-transformer deep learning material it covers in depth. The transformer and scaling material is absent; treat it as a foundational complement to Bishop and Bishop rather than a primary text.
Howard and Gugger’s Deep Learning for Coders with fastai and PyTorch (2020, free with fast.ai courses) takes the opposite approach: top-down, empirical first, with working code from the first chapter. It is the right starting point for learners who want to see things working before understanding why. The fast.ai pedagogical philosophy — build, then understand — complements the mathematical approaches above.
For transformer architecture specifically, Phuong and Hutter’s “Formal Algorithms for Transformers” (arXiv, 2022, free) provides the clearest technical treatment: pseudocode for every operation, precise definitions, and coverage of the major transformer variants. Reading it alongside working code resolves most confusion about implementation details.
For generative models, the original papers are the most efficient entry: Kingma and Welling’s VAE paper (2013), Goodfellow et al.’s GAN paper (2014), Ho, Jain, and Abbeel’s DDPM paper (2020), and Song et al.’s score-matching papers. Each is sufficiently self-contained for a reader with ML foundations.
Roberts, Yaida, and Hanin’s The Principles of Deep Learning Theory (Cambridge, 2022, free at deeplearningtheory.com) covers the theoretical analysis of deep networks using field-theoretic methods. It is demanding but provides the most systematic theoretical treatment available of what deep networks compute and why training succeeds.
| Book | Role | Type |
|---|---|---|
| Bishop & Bishop, Deep Learning: Foundations and Concepts (2024) | Current canonical text | Entry |
| Goodfellow, Bengio & Courville, Deep Learning (free) | Mathematical foundations; pre-2017 architecture | Entry |
| Howard & Gugger, Deep Learning for Coders + fast.ai (free) | Practical top-down entry | Practice |
| Zhang, Lipton, Li & Smola, Dive into Deep Learning (free) | Interactive book with code and math | Entry |
| Roberts, Yaida & Hanin, Principles of Deep Learning Theory (free) | Theoretical foundations | Depth |
| Phuong & Hutter, “Formal Algorithms for Transformers” (free) | Transformer architecture precise reference | Reference |
Courses and Lectures
fast.ai’s Practical Deep Learning for Coders (free at fast.ai) and its companion course on the foundations are the most effective free courses for practitioners. The top-down approach — working code first, mathematical justification as needed — is pedagogically effective for learners with programming backgrounds.
Stanford CS231n (Convolutional Neural Networks for Visual Recognition, notes and slides free at cs231n.github.io) is the canonical course on deep learning for vision, and its treatment of backpropagation, convolutional networks, and training is among the clearest available.
Andrej Karpathy’s Neural Networks: Zero to Hero (free on YouTube) builds a language model from scratch, implementing automatic differentiation, neural network layers, and the transformer architecture step by step. This hands-on sequence produces understanding that reading alone cannot.
Stanford CS224N (Natural Language Processing with Deep Learning, free materials) covers transformer-based NLP with mathematical depth and is the right companion for understanding the architecture from the language modeling perspective.
| Course | Platform | Type |
|---|---|---|
| fast.ai Practical Deep Learning (free) | fast.ai | Entry |
| Stanford CS231n CNN for Visual Recognition (free) | cs231n.github.io | Entry |
| MIT 6.S191 Introduction to Deep Learning (free) | MIT OCW / YouTube | Entry |
| Karpathy, Neural Networks: Zero to Hero (free) | YouTube | Practice |
| Stanford CS224N NLP with Deep Learning (free) | Stanford / YouTube | Entry |
| Full Stack Deep Learning (free) | fullstackdeeplearning.com | Practice |
Practice, Tools, and Current Sources
Andrej Karpathy’s micrograd and nanoGPT (both free on GitHub) are the most instructive implementations in the field. micrograd (~150 lines) implements scalar autodiff; nanoGPT implements a complete GPT-2 training pipeline in ~300 lines. Working through both makes the mechanics of backpropagation and transformer training fully operational.
PyTorch (pytorch.org) is the dominant research framework. Its dynamic computation graph and pythonic interface make it easier to debug and modify than TensorFlow. The official tutorials, particularly the one on building a neural network from scratch, are well-maintained. JAX (Google, free) is preferred in research contexts requiring compilation and hardware portability; understanding JAX’s functional design and XLA compilation is valuable for HPC-oriented deep learning.
The Hugging Face ecosystem (transformers, datasets, accelerate libraries, all free) is the standard infrastructure for working with pre-trained language and vision models. For any work with transformer-based models, fluency with the Hugging Face API is practically necessary.
Weights & Biases (wandb, free tier available) provides experiment tracking, visualization of training curves, and model comparison infrastructure. Systematic experiment tracking — logging hyperparameters, metrics, and model artifacts for every run — is a prerequisite for the reproducibility that serious deep learning requires.
| Resource | Platform | Type |
|---|---|---|
| Karpathy micrograd + nanoGPT (free) | GitHub | Practice |
| PyTorch tutorials (free) | pytorch.org | Practice |
| JAX documentation (free) | jax.readthedocs.io | Practice |
| Hugging Face ecosystem (free) | huggingface.co | Practice |
| Papers With Code 2 (free) | paperswithcode2.com | Reference |
| Weights & Biases (free tier) | wandb.ai | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Framework fluency as deep learning competence | PyTorch makes it possible to train models without understanding what is happening. Practitioners who can call model.fit() but cannot explain what the optimizer is doing, cannot diagnose why training is failing, and cannot evaluate whether the architecture is appropriate for the problem have acquired a useful skill and missed the subject. |
Implement at minimum a two-layer MLP and a small transformer from scratch in NumPy before working with frameworks. Karpathy’s Zero to Hero sequence is the right structured path. Framework fluency should rest on the foundation that understanding provides, not replace it. |
| Architecture cargo-culting | Transformers work remarkably well across many domains, and there is a widespread tendency to apply them to every new problem regardless of whether the all-to-all attention assumption matches the data structure. A transformer on tabular data with 50 features and 1,000 rows is likely to be outperformed by gradient boosting with far less effort. | Before choosing an architecture, ask what structure the data has and which architectural inductive biases fit it. For tabular data: gradient boosting. For small images: CNNs. For long sequences: recurrent or state-space models may outperform transformers. The architecture choice should be motivated, not defaulted. |
| Conflating benchmark improvement with understanding | Deep learning has a systematic problem with benchmarks that measure surface patterns rather than the capabilities they claim. Models can achieve near-human accuracy on benchmarks while failing on trivially modified inputs that any human would handle correctly. “Model X achieves Y on benchmark Z” should prompt the question: what does benchmark Z actually measure? | Supplement quantitative benchmark results with qualitative analysis. Try inputs near the decision boundary. Try distribution-shifted inputs. Try adversarial examples. Read papers that analyze failure modes, not just those that report improvements. |
| Treating scale as a substitute for understanding | It is tempting to believe that scaling a model on a task will eventually solve it, given the success of scaling in language modeling. But scaling is not universal: some capabilities improve smoothly with scale; others show emergent transitions that are unpredictable; others do not improve with scale at all. Compositional generalization, causal reasoning, and systematic generalization have not responded to scale as naively expected. | Understand what scaling laws predict and what they don’t. Study the cases where scaling has and has not worked. The mechanistic interpretability literature is useful here: understanding what networks are actually learning, rather than what we hope they are learning, provides better predictions about when scaling will help. |
| Underestimating data work | The dominant narrative around deep learning focuses on architectures, training procedures, and compute. In production, data quality and composition routinely determine outcomes more than any of these. Models trained on dirty data, imbalanced data, or data with label leakage produce poor results that no architectural sophistication can remedy. | Treat data engineering as primary work, not preparatory work. Spend at least as much time understanding the training data — its distribution, its potential biases, its relationship to the deployment distribution — as on the model. The data-centric AI movement articulates this priority explicitly. |
| Skipping the theory because it “doesn’t matter in practice” | The theory of deep learning — why overparameterized networks generalize, what the implicit bias of gradient descent is, what scaling laws predict — is often dismissed as irrelevant to practitioners. In practice, theoretical understanding prevents the most common systematic errors: overfitting diagnoses that don’t match the double-descent picture, optimizer choices that don’t match the problem, architectural choices that don’t match the data structure. | Engage with the foundational theory in §5.2, the scaling law literature, and the mechanistic interpretability literature. These are not academic luxuries; they are the frameworks that make deep learning failures diagnosable and design choices motivated. |
中文
深度学习系统由许多层可微函数组合而成，这些函数的参数会被共同训练，以最小化数据上的损失。关键在于“深度”：只要宽度足够，神经网络的单层理论上可以计算任何函数，但这可能需要一个指数级庞大的网络。多层结构则会学习一套表征层级——早期层检测简单模式，后续层把它们组合成越来越抽象的模式——这种方式为具有层级结构的数据编码了有用的归纳偏置。对于图像，这种结构是空间性的；对于语言，它是序列性的和组合性的；对于分子生物学，它是几何性的。选择正确的架构，就是为数据中存在的结构编码正确的归纳偏置。
训练过程是第二个定义性要素。反向传播——也就是应用链式法则，在函数组合中计算梯度——使我们可以用梯度下降训练多层网络。这个计算是自动完成的：给定一个可微计算图和一个损失函数，自动微分可以在一次反向传播中计算损失相对于每个参数的梯度。富有表达力的架构、自动微分和大规模数据集结合在一起，产生了能够在感知、生成和推理任务上达到或超过人类水平的系统，而这些任务在几十年里都曾被认为难以处理。
本节直接建立在 §5.2 之上。§5.2 中的统计学习理论、优化概念和评估纪律会贯穿本节。这里新增的是架构词汇、训练技术，以及深度网络特有的经验现象——尤其是围绕过参数化模型泛化能力、规模带来的涌现能力，以及如何设计能编码合适归纳偏置的架构等问题。
前置知识：机器学习基础（§5.2）——默认全部掌握。线性代数（§2.3）——达到矩阵微积分水平。微积分（§2.4）——梯度和链式法则。概率论（§2.5）——用于理解损失函数和生成模型。
从感知机到 Transformer：一门学科的经验发展史
深度学习的故事，是一个好想法不断被重新发现、不断被推进到理论支持之外，并最终被硬件、数据和规模证明有效的故事。
Frank Rosenblatt 的感知机（1957）通过根据错误调整权重来学习分类输入——这是第一台真正能从样本中学习决策规则，并且带有收敛保证的机器。它激发的乐观情绪在 1969 年被 Minsky 和 Papert 的 Perceptrons 打断。该书证明单层网络无法学习 XOR，更重要的是，它似乎还论证了增加层数也无济于事。神经网络研究因此急剧收缩。第一次寒冬并不像传说中那样彻底：Paul Werbos 在 1974 年的博士论文中推导出了反向传播，Linnainmaa 也在 1970 年计算出了导数的反向模式累积。但这些结果没有在正确的时刻连接到正确的研究共同体，深层网络在很大程度上沉寂了十年。
重新连接发生在 1986 年。Rumelhart、Hinton 和 Williams 在 Nature 上发表了 “Learning Representations by Back-propagating Errors”，证明由反向传播训练的多层网络可以解决单层网络无法解决的问题，并且能够发展出捕捉数据结构的内部表征。这篇论文的影响立刻而巨大：它把神经网络正在做的事情，从“学习权重”重新框定为“学习表征”，并使多层网络成为研究的中心对象。
Yann LeCun 迈出了下一步：他追问，在训练开始之前，网络应该被设计成发现什么样的表示结构。对于图像，答案是局部空间结构和平移不变性：边缘、纹理这类模式会出现在不同位置，并且无论出现在哪里，都应该被检测出来。卷积神经网络（CNN）——在每个空间位置应用共享权重滤波器——编码了这个先验。LeCun 的 LeNet（1989）使用 CNN 在美国联邦储备系统中读取手写数字，这个系统可靠到 1990 年代末已经处理了美国约 10% 的支票。这是深度学习第一次工业规模部署，尽管当时并没有被这样认识。
1990 年代，这个领域开始分裂。SVM 有更清晰的理论，并在许多基准上表现强劲；神经网络训练难以调参，并且在更深架构中表现出梯度消失问题。Hochreiter 和 Schmidhuber 的 LSTM（1997）通过引入控制时间中信息流动的门机制，解决了序列数据中的梯度消失问题，使循环网络能够学习长距离依赖。LeCun 证明 CNN 仍然有效。但对于任意问题上的一般深层网络，答案仍然难以把握，这个领域的重心转向了核方法。
第二波浪潮不是由算法创新推动的，而是由数据和硬件达到足够规模推动的。ImageNet 数据集——由 Fei-Fei Li 及其同事从 2009 年起构建，包含 1,000 个类别、120 万张带标签图像——提供了合适规模的基准：它大到足以让手工设计特征触及上限，也大到足以让深层网络学会区分细粒度类别。GPU 提供了计算能力：在 ImageNet 上训练一个深度卷积网络，用 CPU 需要数周，用 GPU 则只需数天。
当 Krizhevsky、Sutskever 和 Hinton 将 AlexNet 提交到 2012 年 ImageNet 竞赛时，它取得了 15.3% 的 top-5 错误率——此前纪录是 26.2%，2012 年其他参赛结果也在 26–30% 区间。这个差距大到毫无歧义：深度学习已经在质上改变了计算机视觉的可能性。两年之内，ImageNet 的所有顶尖参赛方案都变成了深度卷积网络。这个事件重塑了 AI 共同体对深度学习能力的理解。
2012–2017 年是架构快速发展的时期。Graves 等人使用 CTC 损失训练 LSTM 做语音识别，取得了当时的最佳结果。Word2Vec（Mikolov 等，2013）表明，在文本上进行 skip-gram 训练会产生具有语义算术性质的词嵌入——“king - man + woman ≈ queen”——这证明语言的分布式表征确实可以被学习出来。ResNet（He 等，2015）通过引入跳跃连接解决了很深网络中的退化问题，使梯度可以直接通过恒等映射流动，从而让 100 层以上的网络成为可能。批归一化（Ioffe 和 Szegedy，2015）通过在训练期间规范化每层激活，使训练更快、更稳定。GAN（Goodfellow 等，2014）为生成模型引入了对抗训练框架。变分自编码器（Kingma 和 Welling，2013）为表征学习和生成提供了概率框架。Dropout 正则化、用于强化学习的深层网络（Atari，2013），以及序列到序列架构，也都在几年内相继出现。
Vaswani 等人在 “Attention Is All You Need”（2017）中提出的 Transformer 改变了这个领域的结构。自注意力——序列中每个位置都关注所有其他位置——使模型可以捕捉长距离依赖，同时避免了 LSTM 难以并行化的序列计算。Transformer 在大规模训练中更快，在翻译上表现更好，并且泛化到了远远超出机器翻译的任务。BERT（Devlin 等，2018）表明，先在掩码语言建模上预训练一个大型 Transformer，再在下游任务上微调，会显著超过任务特定架构。GPT（Radford 等，2018）表明，在大规模文本语料上进行自回归语言建模，会产生对许多任务都有用的表示。
缩放定律的发现，为当前时代提供了组织原则。Kaplan 等人 2020 年的论文 “Scaling Laws for Neural Language Models” 表明，语言模型在保留数据上的表现，会随着计算量、模型规模和数据集规模按平滑幂律改善——而且没有出现饱和迹象。这不仅是对过去结果的描述，也是对未来结果的定量预测。如果幂律成立，那么用更多数据、更多计算训练更大的模型，就可以可预测地获得更好表现。这个洞见重塑了竞争格局：瓶颈从算法创新转移到了计算资源和数据获取。后续历史——GPT-3、PaLM、Chinchilla、LLaMA，以及当代前沿模型——很大程度上就是拥有不同计算与数据优势的组织沿着这些缩放预测前进的故事。
规模带来了小规模模型中不存在的东西，这是这个领域更令人惊讶的发现之一。上下文学习（in-context learning）——只通过上下文窗口中的少量示例解决任务，而不需要梯度更新——似乎在大约 100 亿到 1000 亿参数之间出现，而且并没有被设计进任何架构。思维链推理——把问题拆成明确的中间步骤会大幅提高准确率——也类似地在规模上出现。这些涌现能力并不能通过小模型行为以任何显然方式预测，其机制也只被部分理解。它们也可能不稳定：某些任务在一个规模上显示出涌现能力，但当评估变得更严格时，表现可能突然下降。
当前深度学习格局是异质的。Transformer 仍然主导序列任务，并已扩展到视觉（Vision Transformers，ViT）、音频、蛋白质结构和多模态场景。状态空间模型（Mamba）在长序列上提供效率优势。扩散模型主导图像、视频和音频生成。混合专家架构可以在不按比例增加推理成本的情况下扩展模型容量。把这些统一起来的，是同一个核心训练循环：可微架构、自动微分，以及在超大数据集上进行随机梯度下降。
架构、训练，以及规模改变了什么
架构即归纳偏置
每一种深度学习架构都编码了关于数据结构的假设；架构假设与数据结构之间是否匹配，决定了网络能否在给定数据量下有效学习。
卷积网络假设空间局部性和平移不变性：应用在位置 (i,j) 的滤波器，和应用在位置 (i+10, j+10) 的滤波器是同一个。这对自然图像和音频频谱图来说是极好的假设，因为模式是局部的，并且意义不随位置变化。但对于表格数据来说，这是一个很差的假设，因为第 1 个位置和第 1001 个位置的特征可能高度相关。它的收益很大：参数数量随滤波器大小而不是输入大小增长，而这种归纳偏置大幅减少了学习所需的数据量。
循环网络假设序列结构：每一步都通过隐藏状态依赖此前所有步骤。LSTM 和 GRU 加入门机制，控制哪些信息可以跨步骤保留下来。这个假设适合语言、语音和时间序列数据，但循环架构本质上是序列性的——每一步都需要前一步的输出——因此难以并行化。
Transformer 假设序列中任意位置都可以以相同成本与任意其他位置相关，模型应当通过注意力权重学习哪些关系重要。自注意力机制会为每个位置计算 query、key 和 value；位置 i 与位置 j 之间的注意力权重，与 exp(qᵢ · kⱼ / √d) 成正比。这种全连接关系富有表达力，也可并行化，但代价昂贵：注意力在序列长度上具有 O(n²) 复杂度。高效注意力变体（线性注意力、稀疏注意力、用于内存效率的 flash attention）仍是活跃的工程研究方向。
图神经网络假设相关结构是由节点和边组成的图，信息通过节点聚合邻居信息而传播。这个假设适合分子性质预测、社交网络分析和知识图谱推理。
因此，架构选择是在训练之前纳入领域知识的主要机制。架构设计已经从启发式手艺，逐渐走向接近工程的形式：感受野大小、表达能力、归纳偏置等架构属性可以被分析，而这些分析会指导设计。
深层训练：梯度流与归一化
训练深层网络要求梯度能够从损失函数向后穿过许多层，并且保持可用。早期深层网络受到梯度消失问题困扰：梯度会随深度指数级缩小，使早期层参数几乎得不到更新。后来，有几种方案成为标准做法。
跳跃连接（残差连接）允许梯度直接通过恒等映射流动，绕过每一层中的非线性变换。ResNet 证明，有了跳跃连接后，100 到 1000 层的网络也可以稳定训练，从而极大扩展了可实现的有效深度。
归一化层——批归一化、层归一化、RMS 归一化——通过规范化激活分布来稳定训练，减少会破坏深层网络训练的内部协变量偏移。批归一化在一个 batch 内归一化；层归一化在单个样本内归一化，因此更适合变长序列。不同架构会使用不同归一化选择，其原因部分来自经验。
初始化很重要：参数初始化的方差应当被选择得足以防止训练开始时梯度爆炸或梯度消失。He 初始化（用于 ReLU 激活）和 Xavier/Glorot 初始化（用于 sigmoid 和 tanh）会根据层宽度缩放初始参数方差。即使架构和数据本来支持训练，糟糕初始化也可能让训练无法成功。
优化器会与架构相互作用。Adam 的每参数自适应学习率对 Transformer 尤其有效，因为不同参数的梯度幅度差异很大。优化器中的权重衰减（L2 正则化）——AdamW 与在损失中加入 L2 正则的 Adam 不同——会因为与自适应学习率的交互方式不同，对泛化产生可测量影响。
规模改变了什么：涌现能力与缩放定律
缩放定律是性能与计算量、模型规模或数据量之间的经验幂律关系。对于语言模型，Kaplan 等人发现，保留文本上的交叉熵损失会随 L ∝ N^{-α} 下降，其中 N 是模型规模，在计算最优训练中指数 α 约为 0.07。Hoffmann 等人（Chinchilla，2022）指出 Kaplan 定律存在一个错误：在给定固定计算预算时，最优策略是用更多数据训练一个较小模型，而不是用较少数据训练一个较大模型，从而修正了最优模型/数据比例。
缩放定律可以作为规划工具：在训练之前，它们可以预测某项计算投入大致会带来多少性能提升。它们也揭示出，在曲线的平滑部分，并不存在阈值或质变转折——改进会继续按幂律进行，而不会饱和。
缩放定律没有捕捉的是涌现能力：某些行为在更大规模上出现，但无法通过小模型插值得到预测。上下文学习、多步算术、思维链推理，以及遵循复杂指令的能力，似乎都会在某些规模阈值之上出现。这些转变在规模与能力的关系上表现为阶跃函数，而不是平滑幂律，并且尚未被完整理解。Wei 等人的 “Emergent Abilities of Large Language Models”（2022）记录了这些转变；后续研究则质疑其中一些涌现能力到底是评估指标造成的伪迹，还是确实存在的能力转变。
这个区别在实践上很重要。如果某种能力是平滑缩放的，那么更多计算通常可靠地有帮助。如果某种能力是不连续涌现的，那么小规模实验无法告诉我们这种能力是否会在更大规模上出现。这使得小模型上的经验研究，在预测前沿模型行为时有一部分信息量不足；这也是这个领域的结构性挑战。
学习这一部分会改变什么
深度学习会改变实践者理解处理非结构化数据的计算系统的方式。
第一个变化是架构直觉：能够把数据结构与架构假设对应起来。面对一个新的学习问题，问题不是“哪个架构最好”，而是“这个数据有什么结构，哪种架构的归纳偏置与这种结构一致？”这个问题是有答案的——图像中的空间局部性，语言中的序列依赖，分子中的图结构。真正内化了架构词汇的实践者，才能认真处理这些问题。
第二个变化是训练熟练度。深度网络可能以许多方式训练失败——梯度爆炸、模式崩塌、表示崩塌、由分布偏移导致的训练—测试差距。这些失败模式会在训练曲线和评估指标中表现出特征性信号；训练过许多网络的实践者会发展出诊断性的模式识别能力：比如“loss 在振荡，可能说明学习率太高”，或者“train loss 在下降，但 val loss 基本不变，可能说明正则化不足导致过拟合”。这种诊断能力来自实践，但其基础是理解训练过程中的每个组件正在做什么。
第三个变化是能够有效阅读研究文献。深度学习论文有典型结构——问题形式化、架构描述、实验结果、消融研究——而阅读它们需要相应词汇，才能判断架构选择是否有动机，基线是否合适，消融是否建立了因果关系。能够阅读一篇新论文并判断它代表真正进展还是基准伪迹的实践者，与只按表面接受结果的实践者处于不同层级。
第四个变化是对规模保持校准过的不确定性。学过缩放定律和涌现能力的实践者知道，什么时候可以相信小规模实验，什么时候不可以：平滑缩放行为可以由小实验很好预测；涌现能力则不行。这种校准既能避免过度依赖可能无法在大规模上成立的小规模结果，也能避免对那些确实能预测大规模行为的小规模实验过度怀疑。
资源
书籍与文本
Bishop 和 Bishop 的 Deep Learning: Foundations and Concepts（2024）是当前最新的基础性教材，具有数学深度，并覆盖包括 Transformer 和扩散模型在内的当代架构。对于现在开始认真学习深度学习的学习者，这是正确的主要参考书。
Goodfellow、Bengio 和 Courville 的 Deep Learning（2016，可在 deeplearningbook.org 免费获取）仍然有价值，因为它严谨处理了数学基础——线性代数、概率、数值计算、信息论——并且深入覆盖了 Transformer 之前的深度学习材料。书中缺少 Transformer 和缩放方面的内容；应把它看作 Bishop 和 Bishop 的基础补充，而不是主要教材。
Howard 和 Gugger 的 Deep Learning for Coders with fastai and PyTorch（2020，可随 fast.ai 课程免费获取）采取了相反路径：自上而下，先经验后理论，从第一章开始就有可运行代码。对于想先看到东西跑起来、再理解原因的学习者，这是合适起点。fast.ai 的教学理念——先构建，再理解——与上面的数学路径形成互补。
对于Transformer 架构本身，Phuong 和 Hutter 的 “Formal Algorithms for Transformers”（arXiv，2022，免费）提供了最清晰的技术处理：每个操作的伪代码、精确定义，以及主要 Transformer 变体的覆盖。把它和可运行代码一起读，可以解决大多数实现细节上的困惑。
对于生成模型，原始论文是最高效的入口：Kingma 和 Welling 的 VAE 论文（2013）、Goodfellow 等人的 GAN 论文（2014）、Ho、Jain 和 Abbeel 的 DDPM 论文（2020），以及 Song 等人的 score-matching 论文。对于具有机器学习基础的读者，每篇论文都足够自洽。
Roberts、Yaida 和 Hanin 的 The Principles of Deep Learning Theory（Cambridge，2022，可在 deeplearningtheory.com 免费获取）使用场论方法分析深度网络理论。它要求很高，但提供了目前最系统的理论处理，用来解释深度网络计算什么，以及为什么训练会成功。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Bishop & Bishop, Deep Learning: Foundations and Concepts (2024) | 当前经典教材 | 入门 |
| Goodfellow, Bengio & Courville, Deep Learning（免费） | 数学基础；2017 前架构 | 入门 |
| Howard & Gugger, Deep Learning for Coders + fast.ai（免费） | 自上而下的实践入口 | 实践 |
| Zhang, Lipton, Li & Smola, Dive into Deep Learning（免费） | 带代码和数学的交互式教材 | 入门 |
| Roberts, Yaida & Hanin, Principles of Deep Learning Theory（免费） | 理论基础 | 深入 |
| Phuong & Hutter, “Formal Algorithms for Transformers”（免费） | Transformer 架构精确参考 | 参考 |
课程与讲座
fast.ai 的 Practical Deep Learning for Coders（fast.ai 免费）及其基础课程，是对实践者最有效的免费课程。自上而下的方法——先写可运行代码，再按需要补数学理由——对有编程背景的学习者非常有效。
Stanford CS231n（Convolutional Neural Networks for Visual Recognition，笔记和 slides 可在 cs231n.github.io 免费获取）是深度学习视觉方向的经典课程，它对反向传播、卷积网络和训练的处理，是目前最清楚的材料之一。
Andrej Karpathy 的 Neural Networks: Zero to Hero（YouTube 免费）从零构建一个语言模型，一步步实现自动微分、神经网络层和 Transformer 架构。这种动手序列会产生单靠阅读无法得到的理解。
Stanford CS224N（Natural Language Processing with Deep Learning，免费材料）以数学深度覆盖基于 Transformer 的 NLP，是从语言建模角度理解架构的合适伴随课程。
| 课程 | 平台 | 类型 |
|---|---|---|
| fast.ai Practical Deep Learning（免费） | fast.ai | 入门 |
| Stanford CS231n CNN for Visual Recognition（免费） | cs231n.github.io | 入门 |
| MIT 6.S191 Introduction to Deep Learning（免费） | MIT OCW / YouTube | 入门 |
| Karpathy, Neural Networks: Zero to Hero（免费） | YouTube | 实践 |
| Stanford CS224N NLP with Deep Learning（免费） | Stanford / YouTube | 入门 |
| Full Stack Deep Learning（免费） | fullstackdeeplearning.com | 实践 |
实践、工具与当前资料
Andrej Karpathy 的 micrograd 和 nanoGPT（均在 GitHub 免费）是这个领域最有教学价值的实现。micrograd（约 150 行）实现了标量自动微分；nanoGPT 用约 300 行实现了完整的 GPT-2 训练流水线。完整读懂二者，会让反向传播和 Transformer 训练的机制变成完全可操作的东西。
PyTorch（pytorch.org）是主导性的研究框架。它的动态计算图和 Python 风格接口，比 TensorFlow 更容易调试和修改。官方教程维护良好，尤其是从零构建神经网络的教程。JAX（Google，免费）则更受需要编译和硬件可移植性的研究场景青睐；理解 JAX 的函数式设计和 XLA 编译，对面向 HPC 的深度学习很有价值。
Hugging Face 生态系统（transformers、datasets、accelerate 等库，均免费）是使用预训练语言模型和视觉模型的标准基础设施。对于任何基于 Transformer 的模型工作，熟悉 Hugging Face API 几乎是实践上的必要条件。
Weights & Biases（wandb，有免费层）提供实验追踪、训练曲线可视化和模型比较基础设施。系统化实验追踪——记录每次运行的超参数、指标和模型产物——是严肃深度学习所要求的可复现性的前提。
| 资源 | 平台 | 类型 |
|---|---|---|
| Karpathy micrograd + nanoGPT（免费） | GitHub | 实践 |
| PyTorch tutorials（免费） | pytorch.org | 实践 |
| JAX documentation（免费） | jax.readthedocs.io | 实践 |
| Hugging Face ecosystem（免费） | huggingface.co | 实践 |
| Papers With Code 2（免费） | paperswithcode2.com | 参考 |
| Weights & Biases（免费层） | wandb.ai | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 把框架熟练度当成深度学习能力 | PyTorch 使训练模型可以在不了解发生了什么的情况下完成。实践者如果只会调用 model.fit()，却不能解释优化器在做什么，不能诊断训练为什么失败，也不能评估架构是否适合问题，那么他只是获得了一项有用技能，却错过了这门学科本身。 |
在使用框架前，至少用 NumPy 从零实现一个两层 MLP 和一个小型 Transformer。Karpathy 的 Zero to Hero 系列是合适的结构化路径。框架熟练度应当建立在理解之上，而不是取代理解。 |
| 架构崇拜式照搬 | Transformer 在许多领域表现惊人，因此人们普遍倾向于把它用于每一个新问题，不管全连接注意力假设是否匹配数据结构。对于只有 50 个特征、1,000 行的表格数据，Transformer 很可能会被梯度提升以少得多的成本超过。 | 选择架构之前，先问数据有什么结构，以及哪种架构的归纳偏置适合它。表格数据：梯度提升。小图像：CNN。长序列：循环模型或状态空间模型可能优于 Transformer。架构选择应当有理由，而不是默认套用。 |
| 把基准提升等同于理解 | 深度学习长期存在一个系统性问题：基准测到的往往是表面模式，而不是它们声称要测量的能力。模型可以在基准上达到接近人类的准确率，却在任何人类都能轻松处理的微小改写输入上失败。“模型 X 在基准 Z 上达到 Y”应该引出的问题是：基准 Z 到底测量了什么？ | 用定量基准结果补充定性分析。尝试接近决策边界的输入。尝试分布偏移输入。尝试对抗样本。阅读分析失败模式的论文，而不只是阅读报告提升的论文。 |
| 把规模当作理解的替代品 | 鉴于语言建模中缩放的成功，人们很容易相信：只要把某个任务上的模型不断放大，它最终就会被解决。但缩放并不是普遍答案：有些能力随规模平滑提升；有些能力出现不可预测的涌现转变；有些能力则完全不随规模改善。组合泛化、因果推理和系统性泛化，并没有像朴素预期那样响应规模。 | 理解缩放定律预测什么、不预测什么。研究那些缩放有效和无效的案例。机制可解释性文献在这里很有用：理解网络实际学到了什么，而不是希望它学到了什么，能够更好地预测什么时候缩放会有帮助。 |
| 低估数据工作 | 关于深度学习的主流叙事集中在架构、训练过程和计算资源上。但在生产环境中，数据质量和数据组成经常比这些更决定结果。用脏数据、不平衡数据或带有标签泄漏的数据训练出来的模型，会产生糟糕结果，再复杂的架构也无法补救。 | 把数据工程视为主要工作，而不是准备工作。至少花和模型同样多的时间理解训练数据——它的分布、潜在偏差、与部署分布的关系。以数据为中心的 AI 运动明确表达了这一优先级。 |
| 因为理论“实践中没用”而跳过理论 | 深度学习理论——为什么过参数化网络能够泛化，梯度下降的隐式偏置是什么，缩放定律预测什么——常被实践者当成无关紧要。但在实践中，理论理解可以防止最常见的系统性错误：与双重下降图景不匹配的过拟合诊断、与问题不匹配的优化器选择、与数据结构不匹配的架构选择。 | 学习 §5.2 的基础理论、缩放定律文献和机制可解释性文献。这些不是学术奢侈品；它们是让深度学习失败可诊断、让设计选择有理由的框架。 |