English
Machine learning is the discipline of making programs improve their behavior through exposure to data rather than through explicit instruction. The distinction sounds simple and its consequences are vast. Every classical program has a specification: a programmer decides what the program should do and writes code that does it. A machine learning program has a different structure — it is given examples of the desired input-output relationship, and it adjusts itself until its behavior matches those examples well enough. The mechanism that does the adjusting, the theory of when it works, and the mathematics that characterizes how well it can possibly work are what machine learning foundations covers.
This section is prerequisite to everything else in Chapter 5. Deep learning (§5.3) is a particular family of machine learning architectures. Reinforcement learning (§5.4) applies the same mathematical machinery to decisions under reward rather than labeled examples. Large language models (§5.5) are trained by machine learning at extraordinary scale. Interpretability (§5.6) is in large part the study of what machine learning produces. None of those subjects can be read with any depth without the foundations here. The relationship to §5.1 (Classical AI) is worth stating: classical AI tried to encode intelligence as rules; machine learning treats intelligence as something that can be extracted from data. The two approaches make different foundational bets, and understanding why the bet on data proved so productive requires understanding what it costs.
Prerequisites: Linear algebra (§2.3) — matrix operations, eigenvalues, SVD. Probability and statistics (§2.5) — distributions, expectations, Bayes’ theorem, MLE. Calculus (§2.4) — gradients, chain rule, the Hessian. Information theory (§3.8) — entropy, KL divergence, cross-entropy.
The Bet That Data Could Replace Rules
The formal origins of machine learning trace to a 1959 paper by Arthur Samuel on checkers. Samuel’s program played games against itself, kept track of which board positions led to wins and which led to losses, and adjusted its evaluation function accordingly. It had no rules for why certain positions were good; it learned which positions correlated with winning. By 1962, the program was beating one of the strongest amateur players in Connecticut. Samuel called the approach “machine learning” — a program that “learns from experience.”
The theoretical underpinning came from Frank Rosenblatt’s perceptron (1957), a simple model of a biological neuron that adjusted its weights based on whether its output matched a target label. Rosenblatt demonstrated that the perceptron learning rule would always converge to a correct classifier if the training data was linearly separable. This was the first proof that a program could learn a decision procedure from examples, with a guarantee. The optimism was short-lived. In 1969, Minsky and Papert published Perceptrons, proving that a single-layer perceptron could not learn the XOR function — not linearly separable. Combined with hardware constraints of the era, the result nearly ended neural network research for a decade.
The field survived by moving in a different direction: instead of asking how neurons might learn, it asked what guarantees could be proved about learning in general. In 1984, Leslie Valiant published “A Theory of the Learnable” in Communications of the ACM. Valiant formulated Probably Approximately Correct (PAC) learning: a concept is PAC-learnable if there exists an algorithm that, given enough examples, will with high probability produce a hypothesis that is close to the true concept — where “close” and “high probability” are both quantified precisely. The PAC framework changed what the field was doing. Theorists could ask: is this problem learnable at all? How many examples are required? The answers depend on the complexity of the hypothesis class — the set of possible models the algorithm considers. For linear classifiers on n dimensions, sample complexity grows roughly as n. For more complex hypothesis classes, it grows with the VC dimension: the size of the largest set of points that the class can “shatter” — assign every possible labeling to. A line in two dimensions can shatter any three points in general position, but cannot shatter four; VC dimension of linear classifiers in 2D is 3. The central result of statistical learning theory — the VC theorem — connects combinatorial geometry to statistics to computation in one result.
While learning theory developed through the 1970s and early 1980s, neural networks were largely dormant. The missing ingredient was a way to train networks with more than one layer — because one layer was provably insufficient for interesting problems, but nobody knew how to extend the perceptron learning rule to deeper networks. In 1986, Rumelhart, Hinton, and Williams published “Learning Representations by Back-propagating Errors” in Nature. Backpropagation computed, for each weight in a multi-layer network, how much the network’s error would change if that weight were nudged slightly. The algorithm had been discovered independently several times before — Linnainmaa in 1970, Werbos in 1974 — but the 1986 paper presented it clearly, connected it to psychological theories of learning, and demonstrated it working on problems the field cared about. Networks trained by backpropagation could learn internal representations: a network trained to detect faces would discover something eye-like in its intermediate layers because that representation turned out to be useful. This demonstrated that multi-layer networks could solve problems that a single layer couldn’t — including XOR. The practical impact was limited for another decade: gradients became vanishingly small in deeper layers, making it nearly impossible to train more than two or three hidden layers.
Through the 1990s, the dominant practical approach was kernel methods, particularly the Support Vector Machine (SVM) developed by Vapnik and colleagues. The SVM’s training objective was elegant: among all hyperplanes that correctly separate the training data, find the one that maximizes the margin — the distance from the boundary to the nearest training points. The kernel trick extended this: certain nonlinear transformations of the data can be computed implicitly without constructing the high-dimensional representation, by evaluating a kernel function k(x, z) between pairs of input points. The Gaussian RBF kernel corresponds to mapping data into an infinite-dimensional feature space, yet the computation is a single exponentiation. SVMs dominated competitive ML benchmarks through the late 1990s and early 2000s. Breiman’s Random Forests (2001) and Friedman’s gradient boosting (1999) achieved comparable results with far less tuning and remain the dominant approach for structured tabular data today.
The deep learning era began with a 2006 paper by Hinton, Osindero, and Teh showing that deep networks could be pre-trained layer by layer using unsupervised methods, then fine-tuned with backpropagation. What actually broke the barrier was data and hardware. The ImageNet dataset (1.2 million labeled images, Deng et al., 2009) gave networks a training set large enough to benefit from depth. GPU computing provided the parallel arithmetic capacity that deep network training requires. In 2012, Krizhevsky, Sutskever, and Hinton trained a deep convolutional network (AlexNet) that reduced ImageNet top-5 error from 26% to 15% — a gap so large it convinced the community that deep learning had changed the field. Within two years, almost every computer vision competition was won by deep convolutional networks.
The theoretical understanding of what deep networks were doing lagged dramatically. The classical bias-variance tradeoff predicted that models with more parameters than training examples would overfit catastrophically. What happened was the opposite: models with many more parameters than data achieved excellent generalization. A 2019 paper by Belkin et al. described double descent: as model complexity increases past the interpolation threshold (where training error hits zero), generalization error first rises then falls again. The classical bias-variance tradeoff is the left half of this curve; the right half, describing over-parameterized models, had no good theoretical explanation for years and remains an active research frontier. Practitioners train enormous models, watch them generalize remarkably well, and the theory is still catching up.
Generalization, Inductive Bias, Optimization, and Evaluation
Generalization: The Central Problem
The central question is always whether learned behavior on training data transfers to new data. Training a model well on training data is not the same as performing well on new data, and the gap between them — the generalization gap — is the central problem of machine learning.
Statistical learning theory (VC dimension, PAC bounds, Rademacher complexity) characterizes when and how well transfer is possible. For neural networks, VC dimension analysis gives vacuous bounds — a network with P parameters has VC dimension at most O(P log P), which means the generalization error bound can be far larger than 1 — yet in practice large networks generalize. The current frontier is understanding which aspects of model behavior actually control generalization. Bartlett et al.’s margin-based bounds, Dziugaite and Roy’s PAC-Bayes experiments, and work on the implicit bias of gradient descent all attempt sharper characterizations. The practical implication is that model selection cannot be done purely from theory; it requires empirical validation on genuinely held-out data.
The bias-variance tradeoff gives an early account: a model that is too simple (high bias) cannot fit the training data; a model too complex (high variance) memorizes training noise rather than pattern. The optimal complexity depends on data quantity. This framing predicted that overfitting would prevent large models from generalizing — a prediction double descent has falsified, without a complete theoretical replacement.
Inductive Bias: The Hidden Design Decision
No learning algorithm is a blank slate. The choice of model family, loss function, optimizer, and regularizer each embeds assumptions about what kinds of solutions are preferred. A linear model assumes the target function is a linear combination of features. A convolutional network assumes translation invariance — which is why it works for images and audio but fails for tabular data where that assumption doesn’t hold. Ridge regression assumes smaller weights are better. Attention in transformers assumes that relevant information in a sequence is not necessarily local.
Feature engineering — the pre-deep-learning art of transforming raw inputs into representations that linear models could use — was explicit inductive bias: a practitioner encoding domain knowledge when rules couldn’t be encoded explicitly. Deep learning largely replaced feature engineering with architecture design, but architecture choices encode equally strong inductive biases, less visibly. Understanding inductive bias is what lets practitioners diagnose model failures systematically: a model that fails on inputs far from the training distribution is likely exhibiting the boundary of its inductive bias — the regime where its assumptions break down.
Optimization: Why SGD Works Better Than the Theory Suggests
A machine learning model is typically trained by minimizing a loss function — a scalar measure of how badly the model’s predictions match the training labels. For linear regression, the minimum has a closed-form solution. For almost everything else, it does not.
Stochastic gradient descent (SGD), introduced by Robbins and Monro in 1951 and adapted to machine learning by Bottou in the 1990s, estimates the gradient on a small randomly-chosen minibatch rather than the full dataset. This makes each update cheaper and introduces noise — and the noise, counterintuitively, often helps: it prevents the optimizer from getting stuck in sharp minima that don’t generalize well. Adam (Kingma and Ba, 2015), combining momentum with adaptive per-parameter learning rates, became the default optimizer for most deep learning work within two years of its publication.
The theoretical guarantee for SGD in convex settings is convergence to the global minimum. For non-convex settings — which is what neural networks give you — the guarantee is much weaker: convergence to a stationary point, which might be a local minimum, a saddle point, or a flat region. In practice, SGD on large networks finds solutions with near-zero training loss that generalize well. The implicit bias of gradient descent toward minimum-norm solutions (Soudry et al.), the tendency of noise to favor flatter minima (Hochreiter and Schmidhuber, Keskar et al.), and the smoothing effect of batch normalization (Ioffe and Szegedy, 2015) together partially explain this. The explanation is not complete. The practical consequence is that optimization hyperparameters matter in ways that are not fully theoretically explained.
Evaluation: Where Most Applied ML Goes Wrong
Train-test splitting is the minimum viable evaluation. Hold out a portion of data before training begins, never touch it during model development, and evaluate the final model exactly once. Every look at test-set performance during development is a decision made partly on the basis of that performance, which is leakage. The reported test accuracy becomes an optimistic estimate of generalization.
The deeper failure modes are subtler. Label leakage occurs when features contain information about the label that wouldn’t be available at prediction time. Data leakage occurs when test examples are duplicates or near-duplicates of training examples. Distribution shift occurs when test data comes from a different distribution than training data — the model performs well on the benchmark and fails in deployment. Benchmarks that become popular enough get solved — not because the task is solved, but because models are specifically optimized for the benchmark, often exploiting artifacts the designers didn’t notice. The appropriate response to strong benchmark performance is skepticism about what the benchmark measures, not celebration of the underlying capability.
What Studying This Changes
The practitioner who has internalized ML foundations thinks about modeling problems differently in four ways.
First, the decomposition of error becomes automatic. Training error, generalization gap, and irreducible noise (Bayes error) are distinct quantities with distinct causes. High training error means the model is too simple or optimization has failed. High generalization gap means the model is too complex or the training set is too small or there is distribution shift. Confusing these diagnoses — and the confusion is common — leads to remedies that don’t work.
Second, model selection becomes principled. The choice between a linear model, a gradient-boosted tree, and a neural network is not a matter of fashion; it is a question of what inductive biases fit the problem structure, what data quantity is available, and what computational constraints apply. Random forests and gradient boosting remain superior to neural networks for tabular data with limited rows. Convolutional networks are appropriate when translation invariance is a real structural property of the data. Transformers are appropriate when long-range dependencies matter and data is abundant. Knowing why changes what you reach for.
Third, the mathematical vocabulary for every subsequent topic in Chapter 5 becomes available. Loss functions, gradients, overfitting, regularization, sample complexity — these terms appear in every paper in deep learning, reinforcement learning, and interpretability. Reading those papers without the foundations means reading their words without their content.
Fourth, the calibrated skepticism that the field’s history demands becomes a working disposition. Machine learning has produced dramatic empirical successes and equally dramatic cases of benchmark inflation, catastrophic deployment failures, and results that didn’t replicate. The practitioner who understands why generalization is hard, what makes evaluation valid, and why the theory lags the empirics is equipped to distinguish genuine progress from artifact.
Resources
Books and Texts
Christopher Bishop’s Pattern Recognition and Machine Learning (2006) remains the best single-volume treatment of the probabilistic view of machine learning. It develops maximum likelihood, Bayesian inference, graphical models, kernel methods, and neural networks from a unified probabilistic perspective with full mathematical depth. Nearly twenty years old, it predates deep learning’s rise, but the foundational material it covers is what deep learning rests on.
Kevin Murphy’s Probabilistic Machine Learning: An Introduction (2022) and Advanced Topics (2023, both free at probml.ai) are the contemporary counterpart. Murphy covers more ground than Bishop, reaches into contemporary deep learning, and is more consistently current. For learners starting now, Murphy is the better primary text; for learners who want the clearest development of the Bayesian perspective, Bishop complements it.
Hastie, Tibshirani, and Friedman’s The Elements of Statistical Learning (2nd ed., 2009, free) covers the classical algorithms — linear methods, trees, boosting, SVMs, model selection — from a statistical perspective, with rigor. This is the appropriate reference for understanding what gradient boosting is doing and why SVMs have the properties they do. James et al.’s An Introduction to Statistical Learning (2nd ed., 2021, free) covers the same territory more accessibly.
Shalev-Shwartz and Ben-David’s Understanding Machine Learning: From Theory to Algorithms (2014) is the best accessible entry into statistical learning theory — PAC learning, VC dimension, sample complexity, the relationship between computational and statistical tractability. For readers who want to understand why learning is possible, not just how to do it, this is the necessary complement to the above texts.
For optimization, Boyd and Vandenberghe’s Convex Optimization (2004, free) is the standard reference for the mathematical theory. Bottou, Curtis, and Nocedal’s survey “Optimization Methods for Large-Scale Machine Learning” (SIAM Review, 2018, free) addresses the stochastic, non-convex regime that deep learning actually operates in.
The Lipton and Steinhardt paper “Troubling Trends in Machine Learning Scholarship” (2018, free) is the clearest articulation of the evaluation and methodology failures the field has generated. Reading it early calibrates expectations about what results in the literature actually mean.
| Book | Role | Type |
|---|---|---|
| Bishop, Pattern Recognition and Machine Learning | Probabilistic ML foundation | Entry |
| Murphy, Probabilistic Machine Learning (2 vols., free) | Contemporary probabilistic ML | Entry |
| Hastie, Tibshirani & Friedman, The Elements of Statistical Learning (free) | Classical algorithms, statistical view | Reference |
| James et al., An Introduction to Statistical Learning (free) | Accessible undergraduate entry | Entry |
| Shalev-Shwartz & Ben-David, Understanding Machine Learning | Statistical learning theory | Depth |
| Mohri, Rostamizadeh & Talwalkar, Foundations of Machine Learning (2nd ed.) | Graduate-level learning theory | Reference |
| Boyd & Vandenberghe, Convex Optimization (free) | Optimization theory standard | Reference |
| Chip Huyen, Designing Machine Learning Systems (paid) | Production ML system design | Depth |
| Lipton & Steinhardt, “Troubling Trends in ML Scholarship” (free) | Evaluation methodology and research hygiene | Auxiliary |
Courses and Lectures
Stanford CS229 (Machine Learning, Andrew Ng, lecture notes free at cs229.stanford.edu) remains the most influential ML course. The lecture notes develop linear regression, logistic regression, SVMs, neural networks, and regularization with full mathematical derivations. Ng’s explanations are consistently the clearest in the field. The course does not cover contemporary deep learning at depth; it is the right foundation for everything that follows.
Caltech CS156 (Learning from Data, Yaser Abu-Mostafa, free lectures on YouTube) develops statistical learning theory more carefully than CS229, with particular strength on the VC dimension, bias-variance tradeoff, and the theory of generalization. The accompanying textbook (also titled Learning from Data) is the right companion.
fast.ai’s Practical Deep Learning for Coders (free at fast.ai) takes the opposite approach from the above: top-down, empirical first. It is the best entry for learners who want to see things working before understanding why. Most practitioners benefit from both orientations; fast.ai first, CS229 and Caltech 156 as foundations, is a productive path for learners with strong practical motivation.
| Course | Platform | Type |
|---|---|---|
| Stanford CS229 Machine Learning (notes free) | cs229.stanford.edu | Entry |
| Caltech CS156 Learning from Data (free) | YouTube | Entry |
| fast.ai as practical ML entry (free) | fast.ai | Entry |
| Google Machine Learning Crash Course (free) | developers.google.com | Entry |
| DataTalksClub Machine Learning Zoomcamp (free) | GitHub / YouTube | Practice |
| DeepLearning.AI Machine Learning in Production (certificate / graded work paid) | DeepLearning.AI / Coursera | Practice |
Practice, Tools, and Current Sources
scikit-learn’s documentation (free) is unusually well-written and serves as both reference and teaching material for classical algorithms. Working through its user guide alongside ESL gives the best connection between theory and implementation. The implementation of a new estimator in scikit-learn style — init, fit, predict, with proper handling of input validation and random state — forces engagement with the abstractions that make ML code reusable.
Implementing linear regression, logistic regression, and a basic neural network from scratch using NumPy — no framework — is the most productive single investment for ML foundations. The gradient computation, the weight update rule, the forward and backward passes: these must become operational rather than abstract before frameworks can be used with understanding rather than incantation.
Andrej Karpathy’s micrograd (GitHub, free, ~150 lines of Python) implements scalar reverse-mode autodifferentiation. Working through it, then extending it to handle simple neural networks, makes the connection between calculus and backpropagation concrete in a way that reading cannot.
Kaggle competitions (specifically the tabular competitions, free) provide the most realistic evaluation practice available. The competitive ranking system creates immediate feedback on whether evaluation methodology was correct: leakage shows up as an implausible public leaderboard score that doesn’t hold on the private set.
| Resource | Platform | Type |
|---|---|---|
| scikit-learn documentation (free) | scikit-learn.org | Practice |
| TensorFlow Playground (free) | playground.tensorflow.org | Auxiliary |
| Made With ML (free) | madewithml.com | Practice |
| NumPy scratch implementations (linear regression, MLP) | Local | Practice |
| Karpathy, micrograd (free) | GitHub | Practice |
| Kaggle tabular competitions (free) | kaggle.com | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Skipping the math and going straight to frameworks | scikit-learn and PyTorch make it possible to train models without understanding what is being optimized or why. The result is practitioners who cannot diagnose failures, who mistake high training accuracy for success, and who cannot adapt methods when the standard pipeline doesn’t work. The cost is invisible until something fails — and in production, failures are often subtle and expensive. | Work through the derivations. Implement linear regression, logistic regression, and a basic neural network from scratch with NumPy before using a framework. The investment is a few weeks; the diagnostic capability it produces lasts for years. |
| Treating deep learning as equivalent to machine learning | Deep learning dominates the contemporary literature and has crowded out other methods in the public imagination. But for tabular data (most of industrial and scientific ML), gradient boosting reliably outperforms neural networks. For small-data regimes, Gaussian processes or Bayesian methods are better calibrated. Knowing only deep learning means consistently reaching for the wrong tool. | Deliberately study classical algorithms — SVMs, random forests, gradient boosting, Gaussian processes — before deep learning. Understanding what these methods do differently, and when they win, is what makes the deep learning chapters meaningful. |
| Evaluating on test data more than once | Every look at test-set performance is a decision made partly on the basis of that performance, which is leakage. The reported test accuracy becomes an optimistic estimate of generalization. In competitions, this is known; in research, it has contributed substantially to results that don’t replicate. | Establish evaluation protocol before training begins. Use a validation set for all development decisions. Touch the test set exactly once, at the end. |
| Confusing benchmark performance with task performance | ImageNet accuracy and GLUE scores measure specific distributions that may or may not correspond to what a model needs to do in deployment. Models routinely achieve near-human benchmark performance while failing on distribution shifts a human would find trivial. | When a benchmark result seems important, ask: what is the test set, how was it collected, is there training contamination, and how does the model perform on inputs the benchmark designers didn’t specifically collect? |
| Treating generalization as a problem that more data always solves | More data helps, but it does not fix the wrong inductive bias, does not eliminate distribution shift, and does not guarantee that a model has learned the right function rather than a correlated proxy. | Learn to distinguish empirically between bias-dominated failure (the model is too simple to fit the training data), variance-dominated failure (the model fits training data but not test data), and distribution shift (the model fits both but fails in deployment). Each has a different remedy. |
中文
机器学习是一门研究如何让程序通过接触数据来改进自身行为的学科，而不是通过显式指令来改进。这个区别听起来简单，但它的后果极其深远。每一个古典程序都有一个规约：程序员决定程序应该做什么，并写出能完成这件事的代码。机器学习程序的结构不同——它接收一组关于期望输入—输出关系的例子，然后调整自身，直到它的行为足够好地匹配这些例子。完成这种调整的机制、它何时有效的理论，以及刻画它最多能做到多好的数学，就是机器学习基础所覆盖的内容。
本节是第五章其他所有内容的前置基础。深度学习（§5.3）是机器学习架构中的一个特定家族。强化学习（§5.4）把同一套数学机制应用于奖励下的决策，而不是带标签的样本。大语言模型（§5.5）是在极大规模上由机器学习训练出来的。可解释性（§5.6）很大程度上研究的是机器学习产生了什么。没有这里的基础，就无法深入阅读这些主题。它与 §5.1（古典 AI）的关系也值得明确说明：古典 AI 试图把智能编码成规则；机器学习则把智能看作某种可以从数据中提取出来的东西。这两种方法押下了不同的基础赌注；而要理解为什么押注数据如此有成效，就必须理解它付出了什么代价。
前置知识：线性代数（§2.3）——矩阵运算、特征值、SVD。概率与统计（§2.5）——分布、期望、贝叶斯定理、MLE。微积分（§2.4）——梯度、链式法则、Hessian 矩阵。信息论（§3.8）——熵、KL 散度、交叉熵。
数据可以取代规则这一赌注
机器学习的形式化起源，可以追溯到 Arthur Samuel 1959 年关于跳棋的一篇论文。Samuel 的程序会与自己对弈，记录哪些棋盘局面会导向胜利、哪些会导向失败，并据此调整它的评估函数。它没有关于某些局面为什么好的规则；它学到的是哪些局面与获胜相关。到 1962 年，这个程序已经能击败康涅狄格州最强的业余棋手之一。Samuel 把这种方法称为“机器学习”——一个“从经验中学习”的程序。
理论支撑来自 Frank Rosenblatt 的感知机（1957）：这是一种简单的生物神经元模型，会根据输出是否匹配目标标签来调整权重。Rosenblatt 证明，如果训练数据是线性可分的，感知机学习规则总会收敛到一个正确分类器。这是第一个证明：程序可以从样本中学习一个决策过程，而且带有保证。但这种乐观并没有持续太久。1969 年，Minsky 和 Papert 出版了 Perceptrons，证明单层感知机无法学习 XOR 函数——因为它不是线性可分的。再加上当时的硬件限制，这个结果几乎让神经网络研究停滞了十年。
这个领域通过转向另一条路线存活下来：它不再追问神经元如何学习，而是追问一般意义上的学习可以证明什么保证。1984 年，Leslie Valiant 在 Communications of the ACM 上发表了 “A Theory of the Learnable”。Valiant 提出了可能近似正确学习（Probably Approximately Correct，PAC）：如果存在一个算法，在给定足够多样本后，能够以高概率产生一个接近真实概念的假设，并且“接近”和“高概率”都被精确定量，那么这个概念就是 PAC 可学习的。PAC 框架改变了这个领域的问题方式。理论家可以问：这个问题到底可不可学习？需要多少样本？答案取决于假设类的复杂度——也就是算法所考虑的可能模型集合。对于 n 维线性分类器，样本复杂度大致随 n 增长。对于更复杂的假设类，它会随 VC 维增长：VC 维是该类能够“打散”（shatter）的最大点集大小，也就是能对这些点实现所有可能标注。二维中的一条直线可以打散任意三个一般位置上的点，但不能打散四个点；因此二维线性分类器的 VC 维是 3。统计学习理论的核心结果——VC 定理——把组合几何、统计和计算连接在同一个结果中。
在 1970 年代到 1980 年代初学习理论发展的同时，神经网络基本处于休眠状态。缺失的关键成分是训练多层网络的方法——因为单层已经被证明不足以解决有趣问题，但没有人知道如何把感知机学习规则扩展到更深的网络。1986 年，Rumelhart、Hinton 和 Williams 在 Nature 上发表了 “Learning Representations by Back-propagating Errors”。反向传播会计算多层网络中每一个权重：如果这个权重被轻微调整，网络误差会改变多少。这个算法此前已经被多次独立发现——Linnainmaa 于 1970 年，Werbos 于 1974 年——但 1986 年这篇论文清楚地呈现了它，把它与心理学学习理论连接起来，并展示了它在这个领域关心的问题上有效。由反向传播训练的网络可以学习内部表征：一个被训练用来检测人脸的网络，会在中间层发现某种类似眼睛的东西，因为这种表征事实证明有用。这表明多层网络可以解决单层网络无法解决的问题，包括 XOR。但它的实践影响又被限制了十多年：在更深层中，梯度会变得极小，使训练超过两三层隐藏层的网络几乎不可能。
整个 1990 年代，主导性的实用方法是核方法，尤其是 Vapnik 及其合作者发展出的支持向量机（SVM）。SVM 的训练目标非常优雅：在所有能够正确分离训练数据的超平面中，找到间隔最大的那个——也就是从决策边界到最近训练点的距离最大。核技巧进一步扩展了这一点：某些对数据的非线性变换，可以不显式构造高维表示，而是通过计算输入点对之间的核函数 k(x, z) 隐式完成。高斯 RBF 核对应于把数据映射到无限维特征空间，但计算上只是一次指数运算。SVM 在 1990 年代末和 2000 年代初主导了机器学习竞赛基准。Breiman 的随机森林（2001）和 Friedman 的梯度提升（1999）以少得多的调参量取得了相当的效果，并且至今仍是结构化表格数据中的主导方法。
深度学习时代始于 Hinton、Osindero 和 Teh 2006 年的一篇论文。论文表明，深层网络可以先用无监督方法逐层预训练，再用反向传播微调。但真正打破障碍的是数据和硬件。ImageNet 数据集（120 万张带标签图像，Deng 等，2009）给了网络一个大到足以从深度中获益的训练集。GPU 计算提供了训练深度网络所需的并行算术能力。2012 年，Krizhevsky、Sutskever 和 Hinton 训练了一个深度卷积网络 AlexNet，把 ImageNet 的 top-5 错误率从 26% 降到 15%——这个差距大到足以让整个社区相信深度学习已经改变了这个领域。两年之内，几乎所有计算机视觉竞赛都被深度卷积网络赢下。
关于深度网络到底在做什么，理论理解严重滞后。经典的偏差—方差权衡预测，当模型参数数量超过训练样本数量时，模型会灾难性过拟合。但实际发生的是相反的：参数数量远多于数据的模型取得了极好的泛化效果。Belkin 等人 2019 年的一篇论文描述了双重下降（double descent）：随着模型复杂度超过插值阈值（训练误差降为零的位置），泛化误差先上升，然后再次下降。经典偏差—方差权衡只是这条曲线的左半边；右半边描述的是过参数化模型，多年来没有良好的理论解释，并且至今仍是活跃研究前沿。实践者训练巨大模型，观察它们表现出惊人的泛化能力，而理论仍在追赶。
泛化、归纳偏置、优化与评估
泛化：核心问题
核心问题始终是：在训练数据上学到的行为，是否能够迁移到新数据上。在训练数据上把模型训练好，并不等于它能在新数据上表现好；二者之间的差距——泛化差距——就是机器学习的核心问题。
统计学习理论（VC 维、PAC 界、Rademacher 复杂度）刻画了迁移何时可能，以及能做到多好。对于神经网络，VC 维分析给出的界常常是空洞的——一个有 P 个参数的网络，其 VC 维最多为 O(P log P)，这意味着泛化误差界可能远大于 1——但在实践中，大网络仍然可以泛化。当前前沿问题是理解模型行为中的哪些方面真正控制泛化。Bartlett 等人的基于间隔的界、Dziugaite 和 Roy 的 PAC-Bayes 实验，以及关于梯度下降隐式偏置的工作，都试图给出更精细的刻画。实践含义是：模型选择不能完全依赖理论；它需要在真正保留出来的数据上进行经验验证。
偏差—方差权衡提供了早期解释：过于简单的模型（高偏差）无法拟合训练数据；过于复杂的模型（高方差）会记住训练噪声，而不是模式。最优复杂度取决于数据数量。这个框架曾预测，过拟合会阻止大模型泛化——而双重下降已经证伪了这一预测，尽管尚未给出完整的理论替代品。
归纳偏置：隐藏的设计决策
没有任何学习算法是一块白板。模型家族、损失函数、优化器和正则化器的选择，都会嵌入关于偏好哪类解的假设。线性模型假设目标函数是特征的线性组合。卷积网络假设平移不变性——这就是为什么它适用于图像和音频，却不适用于表格数据，因为表格数据不满足这个假设。岭回归假设更小的权重更好。Transformer 中的注意力假设：序列中的相关信息不一定是局部的。
特征工程——深度学习之前把原始输入转换成线性模型可用表征的技艺——是一种显式归纳偏置：当规则无法被显式编码时，实践者把领域知识编码进去。深度学习很大程度上用架构设计取代了特征工程，但架构选择同样编码了强大的归纳偏置，只是更不显眼。理解归纳偏置，才能让实践者系统性诊断模型失败：一个模型如果在远离训练分布的输入上失败，很可能就是暴露了它的归纳偏置边界——也就是它的假设开始失效的区域。
优化：为什么 SGD 比理论暗示的更有效
机器学习模型通常通过最小化损失函数来训练——损失函数是一个标量，用来衡量模型预测与训练标签不匹配的程度。对于线性回归，最小值有闭式解。对于几乎其他所有问题，都没有。
随机梯度下降（SGD）由 Robbins 和 Monro 于 1951 年提出，并由 Bottou 在 1990 年代改造用于机器学习。它不是在整个数据集上估计梯度，而是在一个随机选出的小批量样本上估计梯度。这使得每次更新更便宜，也引入了噪声——而这种噪声反直觉地常常有帮助：它防止优化器困在无法良好泛化的尖锐极小值中。Adam（Kingma 和 Ba，2015）把动量和每个参数自适应学习率结合起来，在发表后两年内成为大多数深度学习工作的默认优化器。
SGD 在凸设定中的理论保证，是收敛到全局最小值。对于非凸设定——也就是神经网络给你的设定——保证弱得多：收敛到一个驻点，而它可能是局部最小值、鞍点或平坦区域。在实践中，SGD 在大网络上会找到训练损失接近零、并且泛化良好的解。梯度下降对最小范数解的隐式偏置（Soudry 等）、噪声倾向于偏好更平坦极小值的趋势（Hochreiter 和 Schmidhuber，Keskar 等），以及批归一化的平滑效应（Ioffe 和 Szegedy，2015），共同给出了部分解释。但解释并不完整。实践后果是：优化超参数会以尚未被理论完全解释的方式产生重要影响。
评估：多数应用机器学习出错的地方
训练—测试划分是最低限度的可行评估。在训练开始前保留一部分数据，模型开发过程中绝不触碰它，并且只在最后对最终模型评估一次。开发过程中每看一次测试集表现，都是部分基于该表现做出了一次决策，这就是泄漏。报告出来的测试准确率于是会变成对泛化能力的乐观估计。
更深层的失败模式更微妙。标签泄漏发生在特征包含了预测时不可能获得的标签信息时。数据泄漏发生在测试样本是训练样本的重复或近重复时。分布偏移发生在测试数据来自不同于训练数据的分布时——模型在基准上表现很好，但部署后失败。足够流行的基准最终会被“解决”——不是因为任务被解决了，而是因为模型被专门优化来适应这个基准，常常利用了设计者没有注意到的伪迹。面对强基准表现，合适的回应不是庆祝底层能力，而是怀疑这个基准到底测量了什么。
学习这一部分会改变什么
真正内化机器学习基础的实践者，会以四种不同方式思考建模问题。
第一，误差分解会变成自动反应。训练误差、泛化差距和不可约噪声（贝叶斯误差）是不同量，有不同原因。高训练误差意味着模型太简单，或者优化失败。高泛化差距意味着模型太复杂、训练集太小，或者存在分布偏移。混淆这些诊断——而这种混淆很常见——会导致无效的补救措施。
第二，模型选择会变得有原则。在线性模型、梯度提升树和神经网络之间做选择，不是追逐流行，而是要看什么样的归纳偏置适合问题结构，有多少数据可用，以及有什么计算约束。对于行数有限的表格数据，随机森林和梯度提升仍然优于神经网络。当平移不变性确实是数据的结构性属性时，卷积网络才合适。当长距离依赖重要且数据丰富时，Transformer 才合适。知道原因，会改变你伸手选择的工具。
第三，理解第五章后续所有主题所需的数学词汇会变得可用。损失函数、梯度、过拟合、正则化、样本复杂度——这些术语会出现在深度学习、强化学习和可解释性的每一篇论文中。没有基础就阅读这些论文，等于读到了词，却没有读到内容。
第四，这个领域的历史要求一种校准过的怀疑态度，而这种态度会变成工作习惯。机器学习产生过惊人的经验成功，也产生过同样惊人的基准膨胀、灾难性部署失败和无法复现的结果。理解为什么泛化困难、什么使评估有效、为什么理论滞后于经验的实践者，才有能力区分真正的进展和伪迹。
资源
书籍与文本
Christopher Bishop 的 Pattern Recognition and Machine Learning（2006）仍然是从概率视角处理机器学习的最佳单卷本教材。它从统一的概率视角，以完整的数学深度展开最大似然、贝叶斯推断、图模型、核方法和神经网络。它已有将近二十年历史，早于深度学习崛起，但它覆盖的基础材料正是深度学习所依托的东西。
Kevin Murphy 的 Probabilistic Machine Learning: An Introduction（2022）和 Advanced Topics（2023，均可在 probml.ai 免费获取）是当代对应版本。Murphy 覆盖面比 Bishop 更广，进入了当代深度学习，并且整体上更新。对于现在开始学习的人，Murphy 是更好的主教材；对于想要最清楚理解贝叶斯视角展开方式的学习者，Bishop 是很好的补充。
Hastie、Tibshirani 和 Friedman 的 The Elements of Statistical Learning（第 2 版，2009，免费）从统计视角严谨地覆盖了古典算法——线性方法、树、提升、SVM、模型选择。这是理解梯度提升在做什么、以及 SVM 为什么具有那些性质的合适参考书。James 等人的 An Introduction to Statistical Learning（第 2 版，2021，免费）覆盖同一领域，但更易读。
Shalev-Shwartz 和 Ben-David 的 Understanding Machine Learning: From Theory to Algorithms（2014）是进入统计学习理论的最佳易读入口——PAC 学习、VC 维、样本复杂度，以及计算可处理性与统计可处理性之间的关系。对于想理解学习为什么可能，而不只是怎么操作的读者，这是上述教材的必要补充。
对于优化，Boyd 和 Vandenberghe 的 Convex Optimization（2004，免费）是数学理论的标准参考。Bottou、Curtis 和 Nocedal 的综述 “Optimization Methods for Large-Scale Machine Learning”（SIAM Review，2018，免费）则讨论了深度学习实际运行所在的随机、非凸区域。
Lipton 和 Steinhardt 的论文 “Troubling Trends in Machine Learning Scholarship”（2018，免费）是对这个领域已经产生的评估和方法论失败最清楚的表达。早读这篇文章，可以帮助你校准对文献中结果含义的预期。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Bishop, Pattern Recognition and Machine Learning | 概率机器学习基础 | 入门 |
| Murphy, Probabilistic Machine Learning（两卷，免费） | 当代概率机器学习 | 入门 |
| Hastie, Tibshirani & Friedman, The Elements of Statistical Learning（免费） | 古典算法，统计视角 | 参考 |
| James et al., An Introduction to Statistical Learning（免费） | 易读的本科入门 | 入门 |
| Shalev-Shwartz & Ben-David, Understanding Machine Learning | 统计学习理论 | 深入 |
| Mohri, Rostamizadeh & Talwalkar, Foundations of Machine Learning（第 2 版） | 研究生级学习理论 | 参考 |
| Boyd & Vandenberghe, Convex Optimization（免费） | 优化理论标准参考 | 参考 |
| Chip Huyen, Designing Machine Learning Systems（付费） | 生产级机器学习系统设计 | 深入 |
| Lipton & Steinhardt, “Troubling Trends in ML Scholarship”（免费） | 评估方法论与研究规范 | 辅助 |
课程与讲座
Stanford CS229（Machine Learning，Andrew Ng，讲义可在 cs229.stanford.edu 免费获取）仍然是影响力最大的机器学习课程。讲义以完整数学推导展开线性回归、逻辑回归、SVM、神经网络和正则化。Ng 的解释在这个领域中一直是最清楚的。该课程并不深入覆盖当代深度学习；它是后续所有内容的正确基础。
Caltech CS156（Learning from Data，Yaser Abu-Mostafa，YouTube 免费讲座）比 CS229 更仔细地展开统计学习理论，尤其擅长讲 VC 维、偏差—方差权衡和泛化理论。配套教材也名为 Learning from Data，是合适的伴读材料。
fast.ai 的 Practical Deep Learning for Coders（fast.ai 免费）采取了与上述课程相反的方法：自上而下，先经验后理论。对于想先看到东西跑起来再理解原因的学习者，这是最好的入口。多数实践者会同时受益于这两种方向；对于实践动机强的学习者，先学 fast.ai，再用 CS229 和 Caltech 156 打基础，是一条有效路径。
| 课程 | 平台 | 类型 |
|---|---|---|
| Stanford CS229 Machine Learning（讲义免费） | cs229.stanford.edu | 入门 |
| Caltech CS156 Learning from Data（免费） | YouTube | 入门 |
| fast.ai as practical ML entry（免费） | fast.ai | 入门 |
| Google Machine Learning Crash Course（免费） | developers.google.com | 入门 |
| DataTalksClub Machine Learning Zoomcamp（免费） | GitHub / YouTube | 实践 |
| DeepLearning.AI Machine Learning in Production（证书/评分作业付费） | DeepLearning.AI / Coursera | 实践 |
实践、工具与当前资料
scikit-learn 的文档（免费）写得异常好，既可以作为参考，也可以作为古典算法的教学材料。把它的用户指南和 ESL 一起学习，是连接理论与实现的最佳方式。以 scikit-learn 风格实现一个新的估计器——init、fit、predict，并正确处理输入验证和随机状态——会迫使你接触那些让机器学习代码可复用的抽象。
用 NumPy 从零实现线性回归、逻辑回归和一个基础神经网络——不使用框架——是机器学习基础中最值得投入的一件事。梯度计算、权重更新规则、前向传播和反向传播：这些必须先变成可操作的东西，而不是抽象概念。之后再使用框架，你才是在理解中使用，而不是像念咒一样调用。
Andrej Karpathy 的 micrograd（GitHub，免费，约 150 行 Python）实现了标量反向模式自动微分。完整读懂它，再把它扩展到能处理简单神经网络，可以让微积分和反向传播之间的联系变得具体，这是单靠阅读很难做到的。
Kaggle 竞赛（尤其是表格数据竞赛，免费）提供了最接近真实的评估练习。竞争排名系统会立即反馈你的评估方法是否正确：泄漏通常会表现为一个不合理地高的公开榜分数，但这个分数无法在私有测试集上保持。
| 资源 | 平台 | 类型 |
|---|---|---|
| scikit-learn documentation（免费） | scikit-learn.org | 实践 |
| TensorFlow Playground（免费） | playground.tensorflow.org | 辅助 |
| Made With ML（免费） | madewithml.com | 实践 |
| NumPy scratch implementations（linear regression, MLP） | 本地 | 实践 |
| Karpathy, micrograd（免费） | GitHub | 实践 |
| Kaggle tabular competitions（免费） | kaggle.com | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 跳过数学，直接上框架 | scikit-learn 和 PyTorch 使训练模型可以在不了解优化对象和原理的情况下完成。结果就是实践者无法诊断失败，把高训练准确率误认为成功，并且当标准流程不起作用时无法调整方法。代价在失败之前是不可见的；而在生产环境中，失败往往微妙且昂贵。 | 完整推导。先用 NumPy 从零实现线性回归、逻辑回归和一个基础神经网络，再使用框架。投入只是几周，但它带来的诊断能力会持续多年。 |
| 把深度学习等同于机器学习 | 深度学习主导了当代文献，也挤占了公众想象中其他方法的位置。但对于表格数据——也就是大多数工业和科学机器学习——梯度提升通常可靠地优于神经网络。对于小数据场景，高斯过程或贝叶斯方法具有更好的校准。只懂深度学习，意味着经常会拿错工具。 | 在深度学习之前，有意学习古典算法——SVM、随机森林、梯度提升、高斯过程。理解这些方法有何不同、什么时候会胜出，才会让深度学习章节变得有意义。 |
| 多次在测试数据上评估 | 每看一次测试集表现，都是部分基于该表现做出一次决策，这就是泄漏。报告的测试准确率会变成对泛化能力的乐观估计。在竞赛中，这一点众所周知；在研究中，它也显著促成了无法复现的结果。 | 在训练开始前建立评估协议。所有开发决策都使用验证集。只在最后触碰测试集一次。 |
| 把基准表现等同于任务表现 | ImageNet 准确率和 GLUE 分数衡量的是特定分布，而这些分布未必对应模型在部署中真正需要做的事。模型经常能达到接近人类的基准表现，却在对人类来说很简单的分布偏移上失败。 | 当一个基准结果看起来重要时，追问：测试集是什么，如何收集，是否存在训练污染，以及模型在基准设计者没有专门收集的输入上表现如何。 |
| 以为泛化问题总能靠更多数据解决 | 更多数据有帮助，但它不能修复错误的归纳偏置，不能消除分布偏移，也不能保证模型学到了正确函数，而不是某个相关代理变量。 | 学会用经验方法区分：偏差主导的失败（模型太简单，无法拟合训练数据）、方差主导的失败（模型拟合训练数据，但不能拟合测试数据），以及分布偏移（模型拟合训练集和测试集，但部署中失败）。三者对应不同补救方法。 |