English
Deep learning trains compositions of parameterized functions to construct useful representations and outputs. In an image classifier, early computations may detect local patterns and later computations combine information over larger regions. What is learned depends on the data and objective; a neatly labeled hierarchy of human concepts is not guaranteed.
Nonlinearity matters as much as depth. Several linear transformations with no intervening nonlinearity collapse into one matrix multiplication. Nonlinear layers expand the functions that can be represented. Universal-approximation results for suitable single-hidden-layer networks concern approximation under specified assumptions, not exact computation of every function, efficient representation, or successful learning from finite data. Expressibility, trainability, and generalization are separate questions.
Backpropagation applies the chain rule to compute parameter gradients. An optimizer uses those gradients to update parameters. Keeping these operations separate helps diagnose whether a problem comes from the model, derivative calculation, update rule, data, or evaluation. This section adds architectural and training mechanisms to the learning framework in §5.2.
Prerequisites: Small-model training and evaluation (§5.2), matrix operations (§2.3), and the chain rule (§2.4). Begin with a small multilayer network before studying large architectures.
From Perceptrons to Transformers: The Empirical Development of a Discipline
Early neural-network research investigated how adjustable connections could support recognition and learning. Rosenblatt’s perceptron made a linear decision rule trainable from examples. Its limitations motivated richer representations; they did not settle the capabilities of all multilayer networks. Work continued through periods of reduced attention, including research on differentiation, associative memory, and recurrent systems.
Rumelhart, Hinton, and Williams’s 1986 work helped establish backpropagation as a practical method for learning intermediate representations. The chain rule supplied an efficient way to assign derivatives to many parameters. It did not remove problems caused by saturation, ill-conditioned optimization, limited data, or expensive computation.
Convolutional networks developed from ideas about local receptive fields and shared computations. LeCun and collaborators demonstrated gradient-based learning for handwritten character and document recognition. By the 1990s, these methods had practical applications. Locality and weight sharing reduced the number of independent parameters needed to process an image; their usefulness came from matching a problem’s structure, not from using biologically suggestive terminology.
Recurrent networks reused a state update over time. Long dependencies exposed vanishing and exploding gradients, and LSTM introduced a memory path with gates controlling retention and updates. Kernel methods and other statistical techniques developed alongside this work. The history contains overlapping approaches with different strengths, rather than a complete replacement of one school by another.
Layer-wise pretraining attracted renewed attention in the 2000s. The subsequent expansion of deep learning depended on several changes together: datasets, GPUs, nonlinearities, initialization, regularization, software, and experimental practice. AlexNet’s 2012 ImageNet result demonstrated the effectiveness of one such combination. ImageNet as a collection and the particular classification challenge’s training subset should not be treated as the same dataset size.
Architectures and objectives then diversified. Residual connections made very deep networks easier to optimize; normalization changed activation scales and training behavior. Sequence-to-sequence models and attention improved handling of variable-length inputs and outputs. GANs trained a generator against a discriminator, while variational autoencoders used a latent-variable objective with approximate inference. These are different learning formulations, not merely larger versions of a classifier.
The 2017 transformer used attention in place of recurrence for sequence modeling. During training, known tokens at many positions can be processed in parallel while masks enforce permitted information flow. Autoregressive generation still ordinarily produces successive tokens sequentially. BERT and GPT developed different pretraining objectives and adaptation methods on transformer architectures; transfer learning itself predates both.
Empirical scaling studies measured how loss changes with parameters, training data, and compute. They made resource allocation a quantitative research question. Later work investigated alternatives such as state-space sequence models, sparse mixtures of experts, and diffusion-based generation. Their tradeoffs include memory, communication, latency, data requirements, and which computations the objective requires. No single architecture or scale law explains every successful learning system.
Architectures, Training, and What Scale Changes
Architecture as Inductive Bias
Architecture determines which computations share parameters and how information can move. A convolution applies the same filter at multiple positions. Under suitable stride and boundary conditions, translating the input translates the feature map: equivariance. Invariance would mean leaving the output unchanged. Pooling, padding, downsampling, and the final readout affect which property the complete model has. For digit classification, some positional changes should preserve the label; for locating an object, position must remain recoverable.
Recurrent models summarize preceding inputs in a state, repeatedly applying an update such as hₜ = F(hₜ₋₁, xₜ). LSTM and GRU gates regulate what is retained or replaced. The summary has finite capacity, and ordinary recurrence creates dependencies between time steps that limit training parallelism. The advantage is a state that can be updated as observations arrive.
Attention forms content-dependent weighted combinations. Queries and keys determine weights; values carry the information being combined. Dense self-attention compares pairs of positions, with quadratic pairwise work in sequence length for fixed feature dimensions. Causal or sparse masks restrict available connections. Positional information is necessary when order matters; attention connectivity alone does not specify order.
Graph neural networks aggregate information along edges, using shared operations and often order-independent aggregation over neighbors. This fits problems where relationships are part of the input. The graph definition and aggregation rule determine which distinctions the model can make; message passing does not automatically distinguish every pair of different graphs.
Generative architectures add different objectives. An autoregressive model predicts successive parts; a diffusion model learns to reverse a specified corruption process; a variational autoencoder learns a latent-variable model through a tractable training bound. Select architecture and objective together, then compare data requirements, quality, sampling cost, and the forms of conditioning the task needs.
Training at Depth: Gradient Flow and Normalization
Backpropagation multiplies local derivatives along paths and adds contributions from different paths. A simplified chain with derivative 0.5 at each of twenty stages gives 0.5²⁰ ≈ 0.000000954; with derivative 1.5, the product is about 3325. These scalar examples explain vanishing and exploding gradients. Real networks involve Jacobians, whose directions and interactions matter as well as their magnitudes.
A residual block computes y = x + F(x), so its Jacobian is I + J_F(x). The identity term provides a path that does not pass through every derivative inside F. It helps optimization without guaranteeing that every gradient remains well-scaled. Initialization, activation functions, normalization, and depth still interact.
Normalization methods operate over different axes. Batch normalization uses batch statistics, often also aggregating spatial positions within each channel, and normally uses running statistics at inference. Layer normalization uses selected features within each example, commonly within one token representation. RMS normalization rescales by a root-mean-square statistic without subtracting the mean. These operations affect numerical scale and optimization; “removing internal covariate shift” is not a complete established explanation of all their benefits.
Initialization controls early signal and gradient scales. Glorot-style and He-style choices use layer widths and activation assumptions to choose parameter variance. Saturated activations, extreme scales, or an overly large learning rate can make an otherwise expressive network difficult to train. Diagnose the actual activations and gradients rather than treating one named initializer as a guarantee.
Weight decay also depends on the optimizer. With plain SGD, an L2 penalty produces a proportional shrinkage term under the corresponding convention. With adaptive preconditioning, adding that penalty to the gradient generally differs from directly shrinking parameters. AdamW separates weight decay from the adaptive gradient update. Record optimizer settings, schedules, precision, clipping, and normalization behavior when comparing training runs.
What Scale Changes: Emergent Capabilities and Scaling Laws
Scaling laws are empirical fits for specified model families, data, objectives, and training procedures. A common form separates an irreducible loss level from terms that decrease with parameters or training tokens. The exponents and useful range must be estimated; there is no universal “0.07 per decade” rule for model capability. Lower language-model loss is also not a direct measure of truthfulness, tool-use success, or every downstream task.
Kaplan and colleagues studied these relationships in 2020. Hoffmann and colleagues’ Chinchilla study used a different experimental allocation to estimate how model size and token count should grow together under a fixed training-compute budget. Its recommendations concern that optimization problem. A deployment serving many requests may favor spending more training compute on a smaller model to reduce later inference cost. Data quality, repeated data, hardware efficiency, and post-training can change the practical allocation.
Apparent capability thresholds also depend on measurement. Suppose a toy task requires ten independent correct steps, each with success probability p. All-correct accuracy is p¹⁰: increasing per-step success from 0.8 to 0.9 increases full-task success from about 10.7% to 34.9%. The underlying improvement is smooth, yet the aggregate changes strongly; a small test may record no successes until the model improves. Schaeffer and colleagues showed how metric choices can produce apparent emergence in studied model families. This does not prove that all changes with scale are measurement artifacts.
Evaluate both intermediate performance and completed-task success, report uncertainty, and test extrapolations outside the fitted range. In-context learning has no universal parameter-count threshold. Small-model experiments can reveal mechanisms and reject bad designs, while quantitative predictions about a larger system still need validation. Neither smooth loss curves nor a sudden benchmark increase settles how a capability will behave in deployment.
What Studying This Changes
You should be able to trace a small network’s forward computation, gradients, and parameter updates, and explain which architectural choices share information or parameters. Implementing a small CNN or attention layer is useful when accompanied by checks of dimensions, masks, and gradients.
Run controlled comparisons. To study a residual connection, compare otherwise similar models and record optimization behavior as well as final scores. To study data augmentation, specify which transformations should preserve the target and test whether they do. An ablation supports a claim within its training and evaluation setup; removing a component without retuning may mix that component’s effect with a change in optimization conditions.
Learn to investigate failures through several measurements: training and validation loss, gradient norms, activation scales, data examples, throughput, and memory. A flat validation curve does not uniquely identify a cause. A useful experiment changes a suspected cause and tests a prediction about what should improve.
Resources
Books and Texts
Bishop and Bishop’s Deep Learning: Foundations and Concepts covers mathematical foundations and architectures including transformers and diffusion models. Goodfellow, Bengio, and Courville’s Deep Learning remains useful for optimization, regularization, convolutional networks, and representation learning; its 2016 edition predates the transformer. Dive into Deep Learning and Howard/Gugger’s book provide code alongside explanations.
Use Phuong and Hutter’s “Formal Algorithms for Transformers” when you need precise operations and tensor shapes. For generative models, study one objective and sampling procedure at a time, then read the corresponding VAE, GAN, or diffusion paper. Roberts, Yaida, and Hanin’s The Principles of Deep Learning Theory is a more demanding route into theoretical analysis under specified assumptions; it does not explain every practical training regime.
| Book | Role | Type |
|---|---|---|
| Bishop & Bishop, Deep Learning: Foundations and Concepts (2024) | Architectures and training foundations | Entry |
| Goodfellow, Bengio & Courville, Deep Learning (free) | Mathematical foundations; pre-2017 architecture | Entry |
| Howard & Gugger, Deep Learning for Coders + fast.ai (free) | Practical top-down entry | Practice |
| Zhang, Lipton, Li & Smola, Dive into Deep Learning (free) | Interactive book with code and math | Entry |
| Roberts, Yaida & Hanin, Principles of Deep Learning Theory (free) | Theoretical foundations | Depth |
| Phuong & Hutter, “Formal Algorithms for Transformers” (free) | Transformer architecture precise reference | Reference |
Courses and Lectures
fast.ai starts from working applications. Stanford CS231n develops vision models and training methods, while CS224N connects neural architectures to language tasks. Karpathy’s Neural Networks: Zero to Hero follows a sequence from automatic differentiation to language models. Choose one sequence and implement its central examples; use the other courses when a specific explanation is missing.
| Course | Platform | Type |
|---|---|---|
| fast.ai Practical Deep Learning (free) | fast.ai | Entry |
| Stanford CS231n CNN for Visual Recognition (free) | cs231n.github.io | Entry |
| MIT 6.S191 Introduction to Deep Learning (free) | MIT OCW / YouTube | Entry |
| Karpathy, Neural Networks: Zero to Hero (free) | YouTube | Practice |
| Stanford CS224N NLP with Deep Learning (free) | Stanford / YouTube | Entry |
| Full Stack Deep Learning (free) | fullstackdeeplearning.com | Practice |
Practice, Tools, and Current Sources
Read micrograd to follow scalar differentiation and nanoGPT to inspect a compact language-model training loop. In PyTorch or JAX, practice checking tensor shapes, gradients, and device placement. The frameworks make different design choices; understanding one computation in detail is more useful than memorizing several APIs.
Use model libraries such as Hugging Face Transformers to compare your implementation with an established one. Record data splits, code and model versions, hyperparameters, seeds, metrics, and artifacts; a local log or an experiment-tracking service can both serve this purpose. When reproducing a paper, start from code linked by its authors and check which settings and datasets the reported result used.
| Resource | Platform | Type |
|---|---|---|
| Karpathy micrograd + nanoGPT (free) | GitHub | Practice |
| PyTorch tutorials (free) | pytorch.org | Practice |
| JAX documentation (free) | docs.jax.dev | Practice |
| Hugging Face ecosystem (free) | huggingface.co | Practice |
| Paper authors’ code and project pages | Links in original papers | Reference |
| Weights & Biases (free tier) | wandb.ai | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Equating framework fluency with understanding | Automatic differentiation hides bookkeeping, not the need to specify a valid computation. | Trace one forward and backward pass and check dimensions and gradients. |
| Choosing architecture by popularity | Parameter sharing and information flow may not match the task. | Identify relevant structure and compare a simpler model. |
| Reading a score gain as a mechanism explanation | Several changes in data, optimization, or evaluation can produce the gain. | Use controlled comparisons and examine alternative explanations. |
| Assuming scale guarantees a desired capability | Loss scaling, task metrics, and deployment success are different quantities. | Fit within a stated range and evaluate the capability directly. |
| Treating data work as secondary | Sampling, labels, duplicates, and transformations determine the learning signal. | Inspect data and document the construction of each split. |
| Treating theory as either a full prediction or irrelevant | Theorems explain particular models and assumptions, while experiments assess implementations. | Use theory to formulate a testable expectation and identify its conditions. |
中文
深度学习训练带参数的函数组合，以构造有用的表示和输出。在图像分类器中，较早的计算可能检测局部模式，后续计算再组合更大区域的信息。具体学到什么取决于数据与目标，并不保证形成一套整齐对应人类概念的层次。
非线性与深度同样重要。如果中间没有非线性，多次线性变换可以合并成一次矩阵乘法。非线性层扩展了能够表示的函数。适当单隐层网络的通用逼近结果讨论的是明确假设下的近似，不保证精确计算任意函数、高效表示，或从有限数据成功学习。表示能力、可训练性与泛化是不同问题。
反向传播使用链式法则计算参数梯度，优化器再利用梯度更新参数。分清这两个操作，有助于诊断问题来自模型、导数计算、更新规则、数据还是评估。本节在 §5.2 的学习框架上，增加架构与训练机制。
前置知识：小模型训练与评估（§5.2）、矩阵运算（§2.3）及链式法则（§2.4）。先从小型多层网络开始，再研究大型架构。
从感知机到 Transformer：一门学科的经验发展史
早期神经网络研究探索可调整连接怎样支持识别与学习。Rosenblatt 的感知机使线性决策规则可以从例子中训练。它的局限促使研究者寻找更丰富的表示，并未裁定所有多层网络的能力。在关注度降低的时期，求导、联想记忆与循环系统等研究仍然继续。
Rumelhart、Hinton 与 Williams 的 1986 年工作，帮助确立了用反向传播学习中间表示的实用方法。链式法则提供了高效计算大量参数导数的途径，但并没有消除饱和、病态优化、数据不足或计算昂贵等问题。
卷积网络发展了局部感受野与共享计算的思想。LeCun 及其合作者展示了手写字符和文档识别中的梯度学习，到 1990 年代已有实际应用。局部性与权重共享减少了处理图像所需的独立参数数目；其价值来自匹配问题结构，而不只是使用类似生物学的术语。
循环网络在时间上重复使用状态更新。长距离依赖暴露了梯度消失与爆炸问题，LSTM 则引入由门控制保留与更新的记忆通路。核方法及其他统计技术同时发展。这段历史包含优势不同、彼此重叠的路线，并非一个学派完整取代另一个学派。
逐层预训练在 2000 年代重新吸引了关注。随后深度学习的扩展依靠多种变化共同发生：数据集、GPU、非线性函数、初始化、正则化、软件与实验方法。AlexNet 在 2012 年 ImageNet 竞赛中的结果展示了其中一种组合的有效性。ImageNet 整体图像集合与特定分类竞赛的训练子集，也不能混作同一个数据规模。
之后，架构与目标进一步分化。残差连接使很深的网络更容易优化，归一化改变激活尺度及训练行为；序列到序列模型与注意力改善了可变长输入输出的处理。GAN 让生成器与判别器共同训练；变分自编码器使用带近似推断的潜变量目标。这些是不同的学习设定，并不只是更大的分类器。
2017 年的 transformer 在序列建模中以注意力取代循环。在训练时，许多位置上的已知 token 可以并行处理，同时由掩码限制信息流。自回归生成通常仍需依次产生 token。BERT 与 GPT 在 transformer 架构上发展了不同的预训练目标与适配方法；迁移学习本身早于二者。
经验缩放研究测量损失怎样随参数、训练数据与计算量改变，使资源分配成为定量研究问题。后续研究还探索状态空间序列模型、稀疏混合专家与扩散生成等选择。其取舍包括内存、通信、延迟、数据需求，以及目标要求哪些计算。没有一种架构或缩放定律能解释所有成功的学习系统。
架构、训练，以及规模改变了什么
架构即归纳偏置
架构决定哪些计算共享参数，以及信息怎样流动。卷积在多个位置应用同一个滤波器。在合适步长与边界条件下，平移输入会使特征图相应平移，这叫等变性；不变性则是输出保持不变。池化、填充、下采样和最终读出方式，会影响完整模型具有哪种性质。数字分类应容忍某些位置变化，而物体定位则必须保留可恢复的位置信息。
循环模型用状态概括此前输入，反复执行 hₜ = F(hₜ₋₁, xₜ) 这样的更新。LSTM、GRU 的门调节哪些信息保留或替换。这个摘要的容量有限，普通循环还会产生时间步之间的依赖，限制训练并行；优势是可以随着观测到来持续更新状态。
注意力形成依内容变化的加权组合。查询与键决定权重，值携带被组合的信息。在特征维度固定时，稠密自注意力比较位置对，成对计算量随序列长度平方增长。因果掩码或稀疏掩码限制可用连接。若顺序重要，就需要位置信息；仅有注意力连接并不规定顺序。
图神经网络沿边聚合信息，使用共享操作，并常对邻居采用与排列顺序无关的聚合。这适合关系本身就是输入一部分的问题。图的定义与聚合规则决定模型能表达哪些区别；消息传递不会自动区分任意两张不同的图。
生成架构还引入不同目标。自回归模型预测连续的组成部分；扩散模型学习逆转指定的扰动过程；变分自编码器通过可处理的训练界学习潜变量模型。应一起选择架构与目标，再比较数据需求、质量、采样成本，以及任务需要的条件控制方式。
深层训练：梯度流与归一化
反向传播沿路径相乘局部导数，再累加不同路径的贡献。简化链中，若二十个阶段的导数都为 0.5，乘积为 0.5²⁰ ≈ 0.000000954；若导数为 1.5，乘积约为 3325。这些标量例子解释了梯度消失与爆炸。实际网络涉及 Jacobian，方向及相互作用与大小同样重要。
残差块计算 y = x + F(x)，其 Jacobian 为 I + J_F(x)。单位矩阵项提供了一条不必经过 F 内部每个导数的路径，能帮助优化，但不保证所有梯度都保持合适尺度。初始化、激活函数、归一化与深度仍相互影响。
不同归一化方法处理不同轴。批归一化使用批统计量，常在每个通道内同时汇集空间位置，推断时通常使用运行统计量；层归一化使用单个样本内选定特征的统计量，常作用于一个 token 表示；RMS 归一化按均方根缩放而不减去均值。这些操作影响数值尺度与优化；“消除内部协变量偏移”并不是对其全部收益已经确立的完整解释。
初始化控制早期信号与梯度尺度。Glorot 式、He 式选择根据层宽和激活假设决定参数方差。激活饱和、尺度极端或学习率过大，都可能使具有充分表示能力的网络难以训练。应诊断实际激活与梯度，而不是把某种有名字的初始化当成保证。
权重衰减也取决于优化器。在普通 SGD 中，按相应约定加入 L2 惩罚会产生比例收缩项；采用自适应预条件时，把惩罚加入梯度通常不等于直接缩小参数。AdamW 把权重衰减与自适应梯度更新分开。比较训练运行时，应记录优化器设置、学习率计划、数值精度、裁剪及归一化行为。
规模改变了什么：涌现能力与缩放定律
缩放定律是在指定模型家族、数据、目标与训练程序上的经验拟合。常见形式把不可约损失水平与随参数或训练 token 数下降的项分开。指数和有效范围需要估计，并不存在适用于模型能力的普遍“每个数量级 0.07”规则。更低语言模型损失，也不直接衡量真实性、工具使用成功率或每一种下游任务。
Kaplan 等人在 2020 年研究了这些关系；Hoffmann 等人的 Chinchilla 研究使用不同实验分配，估计在固定训练计算预算下模型规模与 token 数应怎样共同增长。其建议针对这个优化问题。如果部署要服务大量请求，可能值得花更多训练计算在较小模型上，以降低后续推断成本。数据质量、重复数据、硬件效率与后训练都会影响实际分配。
看似突然出现的能力也取决于测量。假设一个玩具任务要求十步全部正确，各步独立且成功概率为 p，那么全对准确率为 p¹⁰：单步成功率从 0.8 增到 0.9，完整任务成功率就从约 10.7% 增到 34.9%。底层改善是平滑的，汇总指标却变化很大；小测试集还可能在模型改善到一定程度前一次成功都观测不到。Schaeffer 及其合作者展示了指标选择怎样在研究的模型家族中产生表观涌现。这不证明所有规模变化都只是测量假象。
应同时评估中间表现与完整任务成功率，报告不确定性，并检验拟合范围外的外推。上下文学习没有普遍适用的参数数量门槛。小模型实验可以揭示机制、排除不良设计，而对更大系统的定量预测仍需验证。平滑损失曲线或突然升高的基准分数，都不能单独裁定能力在部署中怎样表现。
学习这一部分会改变什么
学完后，应能跟踪小网络的前向计算、梯度与参数更新，并解释哪些架构选择共享信息或参数。实现小 CNN 或注意力层时，同时检查维度、掩码与梯度，才更有学习价值。
进行受控比较。研究残差连接时，比较其他条件相近的模型，同时记录优化行为和最终分数；研究数据增强时，说明哪些变换应保持目标，并检验它们是否确实如此。消融实验支持其训练与评估设置内的主张；删除组件却不重新调整设置，可能把组件作用与优化条件变化混在一起。
学会结合多种测量调查失败：训练与验证损失、梯度范数、激活尺度、数据例子、吞吐量和内存。验证曲线不变并不唯一确定原因。有用的实验改变一个怀疑因素，并检验关于哪项表现应改善的预测。
资源
书籍与文本
Bishop 与 Bishop 的 Deep Learning: Foundations and Concepts 涵盖数学基础，以及 Transformer、扩散模型等架构。Goodfellow、Bengio 与 Courville 的 Deep Learning 仍适合学习优化、正则化、卷积网络和表示学习；其 2016 年版本早于 Transformer。Dive into Deep Learning 与 Howard/Gugger 的书则将代码和解释结合起来。
需要核对精确运算和张量形状时，可以查阅 Phuong 与 Hutter 的 “Formal Algorithms for Transformers”。学习生成模型时，一次理解一种目标函数和采样过程，再阅读对应的 VAE、GAN 或扩散模型论文。Roberts、Yaida 与 Hanin 的 The Principles of Deep Learning Theory 提供较深入的理论分析，结论有具体假设，并非涵盖所有实际训练情形。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Bishop & Bishop, Deep Learning: Foundations and Concepts (2024) | 架构与训练基础 | 入门 |
| Goodfellow, Bengio & Courville, Deep Learning（免费） | 数学基础；2017 前架构 | 入门 |
| Howard & Gugger, Deep Learning for Coders + fast.ai（免费） | 自上而下的实践入口 | 实践 |
| Zhang, Lipton, Li & Smola, Dive into Deep Learning（免费） | 带代码和数学的交互式教材 | 入门 |
| Roberts, Yaida & Hanin, Principles of Deep Learning Theory（免费） | 理论基础 | 深入 |
| Phuong & Hutter, “Formal Algorithms for Transformers”（免费） | Transformer 架构精确参考 | 参考 |
课程与讲座
fast.ai 从可运行的应用入手。Stanford CS231n 讲视觉模型和训练方法，CS224N 将神经网络架构与语言任务联系起来。Karpathy 的 Neural Networks: Zero to Hero 从自动微分逐步讲到语言模型。选择一条学习顺序并实现其中的主要例子，遇到解释不足的地方再参考其他课程。
| 课程 | 平台 | 类型 |
|---|---|---|
| fast.ai Practical Deep Learning（免费） | fast.ai | 入门 |
| Stanford CS231n CNN for Visual Recognition（免费） | cs231n.github.io | 入门 |
| MIT 6.S191 Introduction to Deep Learning（免费） | MIT OCW / YouTube | 入门 |
| Karpathy, Neural Networks: Zero to Hero（免费） | YouTube | 实践 |
| Stanford CS224N NLP with Deep Learning（免费） | Stanford / YouTube | 入门 |
| Full Stack Deep Learning（免费） | fullstackdeeplearning.com | 实践 |
实践、工具与当前资料
阅读 micrograd 来跟踪标量微分，阅读 nanoGPT 来检查一个较精简的语言模型训练循环。使用 PyTorch 或 JAX 时，练习核对张量形状、梯度和设备位置。这些框架采取不同的设计；详细理解一次计算，比记住多个 API 更有帮助。
用 Hugging Face Transformers 等模型库与自己的实现进行比较。记录数据划分、代码和模型版本、超参数、随机种子、指标及产物；本地日志或实验跟踪服务都可以承担这一任务。复现论文时，从作者链接的代码入手，并核对论文结果使用的设置和数据集。
| 资源 | 平台 | 类型 |
|---|---|---|
| Karpathy micrograd + nanoGPT（免费） | GitHub | 实践 |
| PyTorch tutorials（免费） | pytorch.org | 实践 |
| JAX documentation（免费） | docs.jax.dev | 实践 |
| Hugging Face ecosystem（免费） | huggingface.co | 实践 |
| 论文作者的代码与项目页 | 原论文中的链接 | 参考 |
| Weights & Biases（免费层） | wandb.ai | 实践 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 把框架熟练当作理解 | 自动求导隐藏了记账工作，却不免除正确规定计算的需要。 | 跟踪一次前向和反向传播，检查维度与梯度。 |
| 按流行程度选架构 | 参数共享与信息流可能不匹配任务。 | 识别相关结构，并与简单模型比较。 |
| 把分数提高当作机制解释 | 数据、优化或评估的多种变化都可能产生提升。 | 进行受控比较，考察其他解释。 |
| 认为规模保证所需能力 | 损失缩放、任务指标与部署成功是不同量。 | 在明确范围内拟合，并直接评估能力。 |
| 把数据工作放在次要位置 | 采样、标签、重复项与变换决定学习信号。 | 检查数据，并记录每种划分的构造。 |
| 把理论当作完整预测或毫无用处 | 定理解释特定模型与假设，实验评估具体实现。 | 用理论形成可检验预期，并明确其条件。 |