English
Machine learning constructs or adapts a model using data. The model may predict a label, estimate a probability, discover structure, generate an example, or choose an action. Learning still requires a program and design decisions: what data is available, which models are considered, what objective is optimized, and how success will be measured.
In supervised learning, examples include target outputs. In unsupervised learning, the learner seeks structure without those task labels. Self-supervised learning creates prediction targets from the data itself, such as a missing word or the next token. Reinforcement learning (§5.4) concerns decisions and reward over time. These distinctions describe the learning signal; they do not determine one particular architecture.
The central separation is between fitting the observed data and performing the intended task on new cases. A classifier can learn which camera photographed an object instead of the object’s relevant properties. Excellent training accuracy would then measure success at exploiting the dataset, without establishing success at the intended task. This section develops generalization, inductive bias, optimization, and evaluation as connected questions.
Prerequisites: Vectors and matrices (§2.3), probability and expectations (§2.5), and derivatives (§2.4). Begin with linear models and small datasets; advanced matrix calculus and learning-theory proofs can follow as needed.
From Learning Programs to Statistical Learning
Learning was part of AI from its early years. Arthur Samuel’s checkers work combined search with methods for improving position evaluation from experience. Rosenblatt’s perceptron supplied a trainable linear decision rule. These projects addressed a practical difficulty: specifying every useful decision by hand becomes expensive, while examples or interaction can supply information about which decisions work.
The perceptron convergence theorem applies to linearly separable training data. XOR supplies a small limitation of a linear threshold classifier: the positive points (0,1) and (1,0) cannot be separated by one straight boundary from (0,0) and (1,1). This is a representational limitation of that model, not a theorem that multilayer networks cannot learn. Minsky and Papert’s analysis influenced debate, while compute, funding, algorithms, and competing approaches also shaped subsequent research.
Statistical learning theory connected prediction to assumptions about sampling and model classes. Vapnik and Chervonenkis studied conditions under which empirical frequencies uniformly approximate probabilities over a class. Valiant’s 1984 PAC framework made accuracy and confidence explicit and connected learning to computational efficiency. “Enough examples” could now be a mathematical question, although its answer depended on the learning setup.
Backpropagation made gradient calculation in multilayer models practical and became widely influential after Rumelhart, Hinton, and Williams’s 1986 work. It has a longer history in differentiation and learning research. The important distinction is between computing a gradient and having a training procedure that succeeds: the objective, initialization, data, architecture, and optimization schedule still matter.
Kernel methods offered nonlinear prediction through inner products in a feature space. A support vector machine balances separation and margin, with soft-margin versions allowing violations. A kernel evaluates a feature-space inner product without necessarily constructing the feature vectors. Computing a Gaussian kernel still requires comparing the input vectors; the method does not make high-dimensional computation or storing many pairwise comparisons free.
Decision trees, bagging, random forests, and boosting developed complementary ways to combine predictors. Bagging can reduce instability by averaging models fitted to resampled data. Boosting adds models to improve a chosen objective, often by fitting information from the remaining errors or gradients. These methods remain useful alternatives to neural networks; their value should be assessed for the actual data and constraints rather than asserted as a universal ranking.
Larger datasets, improved training methods, and parallel hardware expanded what neural networks could learn. Their success also exposed limits of simple rules based only on parameter count. Double-descent experiments showed that test error can rise near interpolation and fall again as capacity increases. This does not falsify the bias–variance identity or guarantee that larger models always improve. It shows why the algorithm, regularization, data distribution, and way capacity is varied belong in the explanation.
Generalization, Inductive Bias, Optimization, and Evaluation
Generalization: The Central Problem
Let R(f) be a predictor’s expected loss under the target distribution, and R̂(f) its average loss on the training sample. Training can reduce the second without reducing the first. The generalization gap compares them, while the total error also depends on the quality of the model class, optimization, label noise, and the target distribution itself. Low training error is one observation, not a complete diagnosis.
A simple PAC bound shows how assumptions do work. Suppose binary labels are noise-free, the correct rule belongs to a finite class H, and examples are drawn independently from the same distribution used for evaluation. A learner choosing a rule consistent with all training labels has error at most ε, with probability at least 1−δ, for 0 < ε, δ < 1 once m ≥ (ln|H| + ln(1/δ))/ε. Intuitively, a rule with error greater than ε is unlikely to survive many independent tests; accounting for every candidate introduces the class-size term. The bound does not apply unchanged to noisy labels, dependent samples, or arbitrary distribution shift.
VC dimension extends this capacity analysis beyond finite classes by asking how many points a class can label in every possible way. Affine half-plane classifiers in two dimensions have VC dimension 3. A capacity measure is not simply the number of stored parameters, and a loose worst-case bound does not predict a particular network’s test error. Margin, norm, stability, and PAC-Bayes analyses examine other properties of learners and solutions.
For squared loss at a fixed input, the bias–variance decomposition separates squared bias of the average predictor, variation between predictors trained on different samples, and conditional outcome noise. It is an identity under the relevant expectations. It does not require that variance grow monotonically with every measure of model size. Regularization and the optimization algorithm can change which solution is selected even when many models fit the sample perfectly.
Inductive Bias: Assumptions Behind Prediction
Inductive bias is a learner’s preference among explanations compatible with the observed data. With only two examples, f(0)=0 and f(1)=1, both f(x)=x and f(x)=x² fit perfectly, but predict different values at x=2. More generally, finite observations leave many possible rules. Assumptions about smoothness, symmetry, sparsity, or simplicity make prediction beyond those observations possible.
Bias enters through representation, model class, loss, regularization, optimization, and data collection. A linear model is linear in its chosen features; adding x² as a feature makes nonlinear dependence on the original input possible. Convolution shares a local computation across positions, supporting translation equivariance under appropriate boundary and stride conditions. It does not make every complete CNN translation-invariant. Attention supplies content-dependent interactions, whose allowed pattern also depends on masking and positional information.
Regularization changes which fits are preferred. In ridge regression, minimizing squared error plus λ‖w‖² penalizes large coefficients; the effect depends on feature scaling. Rescaling one feature changes the coefficient needed to express the same predictor and thus its penalty. Preprocessing and regularization are therefore connected modeling choices. Learned representations reduce some manual feature work, while leaving data construction and architecture choices consequential.
Optimization: Fitting the Training Objective
Optimization adjusts a model to reduce a specified objective. For prediction ŷ = wx and loss L = ½(wx−y)², the gradient is (wx−y)x. At x=2, y=4, and w=1, the loss is 2 and the gradient is −4. A gradient-descent step with learning rate 0.1 gives w=1.4, reducing this example’s loss to 0.72. The calculation explains the update; it supplies no evidence yet about performance on new inputs.
Full-batch gradient descent uses the entire training set for each gradient. Minibatch SGD estimates that gradient using sampled examples, reducing the cost per update. Under uniform sampling its estimate is unbiased for the finite-sample objective, but updates have variance. Batch size, learning rate, momentum, and sampling affect both computation and the optimization path. Adam rescales updates using estimates of gradient moments; it is not automatically preferable for every objective or workload.
Convergence results require assumptions about the objective, gradient estimates, and step sizes. Convexity can support guarantees about global optima; smooth nonconvex analyses often instead bound a measure of stationarity. Neither statement guarantees that a finite training run has reached the desired solution. A zero gradient may occur at a saddle, and good training loss does not establish generalization.
Optimization also selects among solutions. Particular linear-model settings admit precise results about implicit bias toward a norm or margin; those results do not provide one universal explanation of deep learning. Check learning curves, gradient magnitudes, parameter updates, and sensitivity to seeds. A small numerical gradient check is useful when implementing derivatives; it does not validate the dataset or the learning objective.
Evaluation: Measuring Performance on New Data
Design the split around the intended prediction. To predict future observations, use a temporal separation; to generalize to new users or devices, hold out those groups. Randomly splitting individual records can put nearly identical cases from the same source on both sides. Fit preprocessing, feature selection, and hyperparameter choices using training data and validation procedures, leaving the final test set outside those choices.
For a small dataset, cross-validation can help estimate performance and select settings; nested cross-validation separates model selection from its evaluation. Repeatedly using a test score to guide revisions makes that test part of development. Reusing the same fixed test solely for a declared report is a different activity, but adaptive comparisons still require accounting for selection. The important issue is how information from evaluation influences decisions.
Match metrics to errors that matter. If only ten of a thousand items are defective, always predicting “not defective” gives 99% accuracy while detecting none. Precision measures how many flagged cases are true positives; recall measures how many positives were found. Decision thresholds trade these quantities against each other, while calibration asks whether predicted probabilities match observed frequencies. A cost-sensitive decision may need different thresholds even when the probability model is unchanged.
Distinguish leakage from distribution shift. Leakage supplies information that should not be available during development or at prediction time; shift changes the relationship between evaluated and deployed data. Report uncertainty, group performance, relevant baselines, and failure cases. More data helps only insofar as its sampling and labels provide useful information for the target task.
What Studying This Changes
You should be able to describe a learning problem before choosing a library: the prediction time, available inputs, target population, learning signal, loss, and evaluation metric. These choices determine what a successful experiment would demonstrate.
Build a small baseline, inspect its mistakes, and change one justified aspect at a time. High training loss can reflect limited representation, optimization problems, noisy labels, or preprocessing errors. A large evaluation gap can reflect overfitting, sampling differences, or a broken evaluation pipeline. Use evidence to distinguish these explanations instead of treating one curve as a unique diagnosis.
Implement linear or logistic regression once, compare analytical gradients with numerical differences, and reproduce the result with a library. Then deliberately create a leaky split or fit preprocessing on the full dataset and observe the evaluation change. Understanding both the update rule and the evaluation procedure prepares you to judge later deep learning and language-model experiments.
Resources
Books and Texts
For a first pass, An Introduction to Statistical Learning connects models to worked examples. Bishop’s Pattern Recognition and Machine Learning and Murphy’s Probabilistic Machine Learning develop the probabilistic formulation in more depth. The Elements of Statistical Learning is useful when comparing classical methods and their assumptions; these books need not all be read in sequence.
For learning theory, use Shalev-Shwartz and Ben-David’s Understanding Machine Learning to connect hypothesis classes, sample complexity, and generalization. For optimization, Boyd and Vandenberghe’s Convex Optimization gives the assumptions behind convex guarantees. Lipton and Steinhardt’s “Troubling Trends in Machine Learning Scholarship” is a useful companion when reading papers: distinguish an observed improvement, a proposed explanation, and evidence that tests that explanation.
| Book | Role | Type |
|---|---|---|
| Bishop, Pattern Recognition and Machine Learning | Probabilistic ML foundation | Entry |
| Murphy, Probabilistic Machine Learning (2 vols., free) | Contemporary probabilistic ML | Entry |
| Hastie, Tibshirani & Friedman, The Elements of Statistical Learning (free) | Classical algorithms, statistical view | Reference |
| James et al., An Introduction to Statistical Learning (free) | Accessible undergraduate entry | Entry |
| Shalev-Shwartz & Ben-David, Understanding Machine Learning | Statistical learning theory | Depth |
| Mohri, Rostamizadeh & Talwalkar, Foundations of Machine Learning (2nd ed.) | Graduate-level learning theory | Reference |
| Boyd & Vandenberghe, Convex Optimization (free) | Optimization theory standard | Reference |
| Chip Huyen, Designing Machine Learning Systems (paid) | Production ML system design | Depth |
| Lipton & Steinhardt, “Troubling Trends in ML Scholarship” (free) | Evaluation methodology and research hygiene | Auxiliary |
Courses and Lectures
Stanford CS229 provides derivations and exercises for core algorithms. Caltech CS156: Learning from Data emphasizes generalization and model selection. Learners who prefer to start with runnable models can use fast.ai, then return to the mathematical topics needed to explain their experiments. Choose based on your prerequisites and preferred order of theory and practice.
| Course | Platform | Type |
|---|---|---|
| Stanford CS229 Machine Learning (notes free) | cs229.stanford.edu | Entry |
| Caltech CS156 Learning from Data (free) | YouTube | Entry |
| fast.ai as practical ML entry (free) | fast.ai | Entry |
| Google Machine Learning Crash Course (free) | developers.google.com | Entry |
| DataTalksClub Machine Learning Zoomcamp (free) | GitHub / YouTube | Practice |
| DeepLearning.AI Machine Learning in Production (certificate / graded work paid) | DeepLearning.AI / Coursera | Practice |
Practice, Tools, and Current Sources
Use the scikit-learn documentation to connect estimators, preprocessing, pipelines, and cross-validation. Implement one linear or logistic regression model with NumPy and compare its predictions and loss with a library implementation. micrograd is a small automatic-differentiation implementation that helps explain where gradients come from; implementing every algorithm from scratch is unnecessary.
A tabular competition can provide practice with validation and feature construction, but leaderboard feedback can itself cause overfitting. Keep a record of the decisions made from each score. A good leaderboard result does not establish that your split is free of leakage or that the model will work on a different population.
| Resource | Platform | Type |
|---|---|---|
| scikit-learn documentation (free) | scikit-learn.org | Practice |
| TensorFlow Playground (free) | playground.tensorflow.org | Auxiliary |
| Made With ML (free) | madewithml.com | Practice |
| NumPy scratch implementations (linear regression, MLP) | Local | Practice |
| Karpathy, micrograd (free) | GitHub | Practice |
| Kaggle tabular competitions (free) | kaggle.com | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Learning only framework calls | An API hides the loss, gradient, sampling, and evaluation choices. | Implement one small model and trace its objective and update. |
| Equating machine learning with deep learning | Trees, linear models, kernels, and probabilistic methods offer different assumptions and costs. | Compare simple baselines under the same split, metric, and budget. |
| Using test results to tune the model | Adaptive choices exploit information from the test and bias later assessment. | Use validation for selection and reserve an independent final assessment. |
| Treating a benchmark score as task success | The dataset, metric, and prediction conditions may differ from the intended use. | Specify the target use and inspect relevant errors and groups. |
| Assuming more data always fixes generalization | Repeated bias, leakage, or irrelevant samples can persist at larger scale. | Check how examples were collected and what new information they add. |
中文
机器学习利用数据构造或调整模型。模型可以预测标签、估计概率、发现结构、生成样本，或选择动作。学习仍然需要程序与设计决定：能够使用什么数据，考虑哪些模型，优化什么目标，以及怎样衡量成功。
监督学习的例子带有目标输出；无监督学习在没有这类任务标签的情况下寻找结构；自监督学习从数据本身构造预测目标，例如被遮住的词或下一个 token。强化学习（§5.4）涉及随时间展开的决策与奖励。这些区别描述学习信号，并不指定某一种架构。
核心区分是拟合已观测数据与在新案例上完成预期任务。分类器可能学会识别拍摄物体的相机，而没有学到物体的相关性质。此时，极高训练准确率说明模型善于利用数据集，却没有确立它能完成预期任务。本节把泛化、归纳偏置、优化与评估作为相互关联的问题展开。
前置知识：向量与矩阵（§2.3）、概率与期望（§2.5）、导数（§2.4）。可以先从线性模型和小数据集开始，再按需补充高级矩阵微积分与学习理论证明。
从学习程序到统计学习
学习从 AI 早期就是其中一条路线。Arthur Samuel 的跳棋研究把搜索与根据经验改善局面评价的方法结合起来；Rosenblatt 的感知机提供了可以训练的线性决策规则。这些项目处理一个实际困难：逐条手写有用决策的成本很高，而例子或交互可以提供哪些决策有效的信息。
感知机收敛定理适用于线性可分的训练数据。XOR 展示了线性阈值分类器的一项小而明确的局限：无法用一条直线，把正例 (0,1)、(1,0) 与 (0,0)、(1,1) 分开。这是该模型的表示能力限制，并不证明多层网络无法学习。Minsky 和 Papert 的分析影响了讨论，计算资源、资助、算法及竞争路线也共同塑造了后续研究。
统计学习理论把预测与采样假设、模型类别联系起来。Vapnik 与 Chervonenkis 研究经验频率在整个函数类上统一逼近概率的条件；Valiant 在 1984 年提出的 PAC 框架明确表达准确程度与置信程度，并把学习连接到计算效率。“需要多少例子”由此成为可以精确提问的问题，但答案取决于学习设置。
反向传播使多层模型中的梯度计算变得实用，并在 Rumelhart、Hinton 与 Williams 的 1986 年工作后得到广泛传播。它在求导和学习研究中有更早的历史。重要区别是：算出梯度与训练成功是两件事；目标、初始化、数据、架构及优化计划仍然重要。
核方法通过特征空间内积实现非线性预测。支持向量机考虑分类分离与间隔，软间隔版本允许违例。核函数可以不显式构造特征向量就计算其内积，但计算高斯核仍需比较输入向量；它并没有使高维计算或大量两两比较的存储变得免费。
决策树、装袋、随机森林和提升方法提供了组合预测器的其他方式。装袋对重采样数据上拟合的模型取平均，可以降低不稳定性；提升逐步加入模型以改善选定目标，常利用剩余误差或梯度的信息。这些方法仍是神经网络之外的有用选择，应根据实际数据与约束评估，而不作普遍排名。
更大的数据集、改进的训练方法与并行硬件扩展了神经网络能学习的内容，其成功也显示了仅凭参数数量判断的局限。双重下降实验表明，测试误差可能在插值附近升高，再随容量增加而下降。这没有推翻偏差—方差恒等式，也不保证更大模型总会改善；它说明解释中需要包含算法、正则化、数据分布及改变容量的方式。
泛化、归纳偏置、优化与评估
泛化：核心问题
让 R(f) 表示预测器在目标分布上的期望损失，R̂(f) 表示它在训练样本上的平均损失。训练可以降低后者而不降低前者。泛化差距比较这两个量；总误差还取决于模型类别是否合适、优化、标签噪声及目标分布本身。低训练误差是一项观测，并非完整诊断。
一个简单 PAC 界可以展示假设怎样发挥作用。假设二元标签无噪声，正确规则属于有限类 H，例子独立抽取，而且来自评估使用的同一分布。如果学习器选择与所有训练标签一致的规则，那么对 0 < ε, δ < 1，当 m ≥ (ln|H| + ln(1/δ))/ε 时，它以至少 1−δ 的概率具有至多 ε 的错误率。直觉是：错误率大于 ε 的规则很难通过许多次独立检验；把所有候选都计入，就出现了类别大小这一项。这个界不能原样用于有噪声标签、依赖样本或任意分布变化。
VC 维通过询问一个函数类能对多少点实现所有可能标记，把容量分析扩展到无限函数类。二维仿射半平面分类器的 VC 维为 3。容量度量不简单等于存储的参数数目，宽松的最坏情况界也不能预测某个网络的测试误差。间隔、范数、稳定性与 PAC-Bayes 分析考察学习器和解的其他性质。
在固定输入处，平方损失的偏差—方差分解区分平均预测器的平方偏差、不同训练样本所产生预测器之间的变化，以及结果的条件噪声。它是在相应期望下成立的恒等式，并不要求方差随每一种模型规模度量单调增长。即使许多模型都能完美拟合样本，正则化与优化算法也会改变最终选中哪个解。
归纳偏置：预测所依赖的假设
归纳偏置是学习器在多种符合观测数据的解释之间表现出的偏好。只有 f(0)=0、f(1)=1 两个例子时，f(x)=x 与 f(x)=x² 都能完美拟合，却对 x=2 给出不同预测。一般来说，有限观测会留下许多可能规则。关于平滑性、对称性、稀疏性或简单性的假设，使超出这些观测的预测成为可能。
偏置通过表示、模型类、损失、正则化、优化及数据收集进入学习过程。线性模型相对于选定特征是线性的；加入 x² 特征，就能表达对原输入的非线性依赖。卷积在不同位置共享局部计算，在合适边界与步长条件下支持平移等变性，但不使每个完整 CNN 都具备平移不变性。注意力提供依内容变化的交互，其允许模式还取决于掩码与位置信息。
正则化改变哪些拟合结果更受偏好。岭回归最小化平方误差加 λ‖w‖²，惩罚较大系数；效果取决于特征尺度。缩放一个特征会改变表达同一预测器所需的系数，因此也改变惩罚。预处理与正则化是相互关联的建模选择。学习表示减少了部分手工特征工作，但数据构造与架构选择仍然重要。
优化：拟合训练目标
优化调整模型以降低指定目标。对于预测 ŷ = wx 和损失 L = ½(wx−y)²，梯度为 (wx−y)x。在 x=2、y=4、w=1 时，损失为 2，梯度为 −4。以学习率 0.1 做一步梯度下降，得到 w=1.4，使这个例子的损失降为 0.72。计算解释了更新过程，尚未提供新输入上表现如何的证据。
全批量梯度下降每次用全部训练集计算梯度；小批量 SGD 用抽取的例子估计梯度，减少每次更新的成本。在均匀采样下，这个估计对有限样本目标无偏，但更新具有方差。批量大小、学习率、动量与采样方式共同影响计算和优化路径。Adam 利用梯度矩的估计缩放更新，并不自动适合每一种目标和工作负载。
收敛结论需要对目标、梯度估计与步长作出假设。凸性可以支持关于全局最优解的保证；光滑非凸分析则常约束某种驻点性度量。两者都不保证有限训练已经达到所需解。梯度为零可能出现在鞍点，良好的训练损失也不确立泛化。
优化还会在多个解之间作出选择。特定线性模型设置允许精确分析对范数或间隔的隐式偏好，但这些结果不构成解释所有深度学习的统一答案。应检查学习曲线、梯度大小、参数更新及对随机种子的敏感性。实现导数时，小型数值梯度检查很有用，但它不会验证数据集或学习目标是否合适。
评估：衡量新数据上的表现
按照预期预测任务设计划分。预测未来观测时使用时间分隔；希望泛化到新用户或新设备时，保留完整的用户或设备组。随机拆分单条记录，可能把来自同一来源的近乎相同案例放到两边。预处理、特征选择和超参数选择应使用训练数据与验证程序，最终测试集不参与这些选择。
小数据集可以用交叉验证估计表现、选择设置；嵌套交叉验证把模型选择与对选择结果的评估分开。反复依据测试分数修改模型，会使测试成为开发的一部分。仅为预先声明的报告而重复使用同一个固定测试，属于不同活动，但自适应比较仍需处理选择影响。重要的是评估信息怎样进入决定。
指标应对应关心的错误。如果一千件物品中只有十件缺陷品，总预测“无缺陷”就有 99% 准确率，却一件也检不出。精确率衡量被标记的案例中有多少是真正正例；召回率衡量全部正例中找到了多少。决策阈值在二者之间产生取舍，校准则询问预测概率是否对应观测频率。即使概率模型不变，对错误成本敏感的决策也可能需要不同阈值。
还要区分泄漏与分布变化。泄漏提供了开发或预测时本不应获得的信息；分布变化则改变评估数据与部署数据之间的关系。报告不确定性、分组表现、相关基线和失败案例。新增数据只有在采样与标签确实提供目标任务所需信息时，才有帮助。
学习这一部分会改变什么
学完后，应能在选择库之前描述学习问题：何时预测、可用输入、目标总体、学习信号、损失及评估指标。这些选择决定成功的实验能够说明什么。
先建立小型基线，检查错误，再一次修改一个有依据的因素。高训练损失可能来自表示能力不足、优化问题、标签噪声或预处理错误；较大的评估差距可能来自过拟合、采样差异或评估流程问题。应使用证据区分这些解释，而不是把一条曲线当作唯一诊断。
亲自实现一次线性回归或逻辑回归，用数值差分核对解析梯度，再用库复现实验。随后故意制造泄漏划分，或在全部数据上拟合预处理，观察评估怎样改变。既理解更新规则，也理解评估程序，才能判断后续深度学习与语言模型实验。
资源
书籍与文本
初读可以用 An Introduction to Statistical Learning，将模型与具体例子联系起来。Bishop 的 Pattern Recognition and Machine Learning 和 Murphy 的 Probabilistic Machine Learning 更深入地展开概率建模。比较经典方法及其假设时，可以查阅 The Elements of Statistical Learning；不必依次读完所有这些书。
学习理论可以用 Shalev-Shwartz 与 Ben-David 的 Understanding Machine Learning，把假设类、样本复杂度与泛化联系起来。优化方面，Boyd 与 Vandenberghe 的 Convex Optimization 解释凸优化保证所需的假设。阅读论文时，可以配合 Lipton 与 Steinhardt 的 “Troubling Trends in Machine Learning Scholarship”：区分观察到的改进、为改进提出的解释，以及检验这个解释的证据。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Bishop, Pattern Recognition and Machine Learning | 概率机器学习基础 | 入门 |
| Murphy, Probabilistic Machine Learning（两卷，免费） | 当代概率机器学习 | 入门 |
| Hastie, Tibshirani & Friedman, The Elements of Statistical Learning（免费） | 古典算法，统计视角 | 参考 |
| James et al., An Introduction to Statistical Learning（免费） | 易读的本科入门 | 入门 |
| Shalev-Shwartz & Ben-David, Understanding Machine Learning | 统计学习理论 | 深入 |
| Mohri, Rostamizadeh & Talwalkar, Foundations of Machine Learning（第 2 版） | 研究生级学习理论 | 参考 |
| Boyd & Vandenberghe, Convex Optimization（免费） | 优化理论标准参考 | 参考 |
| Chip Huyen, Designing Machine Learning Systems（付费） | 生产级机器学习系统设计 | 深入 |
| Lipton & Steinhardt, “Troubling Trends in ML Scholarship”（免费） | 评估方法论与研究规范 | 辅助 |
课程与讲座
Stanford CS229 提供核心算法的推导和练习。Caltech CS156: Learning from Data 着重讲泛化与模型选择。偏好先运行模型的学习者可以从 fast.ai 开始，再补上解释实验所需的数学内容。根据自己的基础和理论、实践的学习顺序选择即可。
| 课程 | 平台 | 类型 |
|---|---|---|
| Stanford CS229 Machine Learning（讲义免费） | cs229.stanford.edu | 入门 |
| Caltech CS156 Learning from Data（免费） | YouTube | 入门 |
| fast.ai as practical ML entry（免费） | fast.ai | 入门 |
| Google Machine Learning Crash Course（免费） | developers.google.com | 入门 |
| DataTalksClub Machine Learning Zoomcamp（免费） | GitHub / YouTube | 实践 |
| DeepLearning.AI Machine Learning in Production（证书/评分作业付费） | DeepLearning.AI / Coursera | 实践 |
实践、工具与当前资料
用 scikit-learn 文档把估计器、预处理、流水线与交叉验证联系起来。用 NumPy 实现一个线性回归或逻辑回归模型，并与库实现比较预测和损失。micrograd 是一个小型自动微分实现，有助于理解梯度从哪里来；不必从头实现每一种算法。
表格数据竞赛可以练习验证和特征构造，但排行榜反馈本身也可能造成过拟合。记录自己根据每次分数做了哪些决定。排行榜成绩好，并不能证明数据划分没有泄漏，也不能证明模型适用于另一类人群或数据来源。
| 资源 | 平台 | 类型 |
|---|---|---|
| scikit-learn documentation（免费） | scikit-learn.org | 实践 |
| TensorFlow Playground（免费） | playground.tensorflow.org | 辅助 |
| Made With ML（免费） | madewithml.com | 实践 |
| NumPy scratch implementations（linear regression, MLP） | 本地 | 实践 |
| Karpathy, micrograd（免费） | GitHub | 实践 |
| Kaggle tabular competitions（免费） | kaggle.com | 实践 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 只学习框架调用 | API 隐藏了损失、梯度、采样与评估选择。 | 实现一个小模型，跟踪目标和更新。 |
| 把机器学习等同于深度学习 | 树、线性模型、核与概率方法具有不同假设和成本。 | 在相同划分、指标和预算下比较简单基线。 |
| 利用测试结果调模型 | 自适应选择会利用测试信息，使后续评估偏乐观。 | 用验证进行选择，保留独立的最终评估。 |
| 把基准分数当作任务成功 | 数据集、指标与预测条件可能不同于预期用途。 | 说明目标用途，检查相关错误和群体。 |
| 认为更多数据总能修复泛化 | 重复的偏差、泄漏或无关样本可以随规模扩大而持续。 | 检查例子的收集方式，以及新增了什么信息。 |