English
Probability describes uncertainty within a model; statistics uses observations to estimate, compare, and check models or claims about a population. The distinction is useful even though the activities interact. Given a coin’s head probability, probability theory describes sequences of tosses. Given a sequence, statistical inference asks what it supports about the coin—and whether independent tosses with a fixed probability are an adequate model.
Uncertainty has several sources: deliberate randomization, noisy measurements, limited samples, and incomplete knowledge. Interpreting a probability as a long-run frequency or as a degree of belief leads to different methodological choices, but these are not different arithmetic rules for probability. The interpretation and the assumptions should travel with the number.
Discrete examples need algebra and counting; continuous distributions introduce integration; multivariable models benefit from linear algebra. Start with small examples and simulations, then connect them to definitions. A simulator can check a derivation under its programmed assumptions; it cannot establish that those assumptions describe the population of interest.
From Gambling Tables to the Mathematics of Learning
Games of chance helped stimulate mathematical probability, with the Pascal–Fermat correspondence of 1654 a familiar landmark. But counting practices, insurance, demography, and measurement also shaped the subject. The law of large numbers connected repeated observations to stable frequencies under specified conditions; work on errors and estimation addressed how imperfect measurements could support reliable conclusions.
Twentieth-century statistics developed several approaches to inference. Fisher emphasized likelihood and significance testing; Neyman and Pearson developed tests with controlled error rates; Bayesian methods used prior distributions and conditioning. These approaches ask related but different questions. Kolmogorov’s 1933 axiomatization provided a common mathematical foundation for probability, not a verdict that only one interpretation or inference method was legitimate.
Computers changed what could be calculated. Simulation, resampling, and Markov chain Monte Carlo made many models usable when exact summation or integration was impractical. They introduced computational questions alongside statistical ones: does a chain explore the relevant distribution, are effective samples numerous enough, and is the estimate stable? More computing does not automatically overcome poor mixing or an inadequate model.
AI’s history adds a useful lesson. Expert systems sometimes attached informal confidence scores to rules, but combining uncertainty coherently required more than propagating arbitrary numbers. Probabilistic graphical models made dependencies explicit and supplied rules for inference. Statistical learning also grew from pattern recognition and estimation, not simply from a wholesale replacement of logic by neural networks.
Information theory supplied another connection. Shannon’s 1948 work related uncertainty to communication and coding; Kullback and Leibler’s 1951 paper developed the divergence bearing their names. These ideas help explain both compression and learning objectives, provided we identify the distributions being compared.
Conditioning, Estimation, and the Measurement of Uncertainty
Keep three questions distinct: what changes when evidence is observed, what can be inferred from a sample, and how uncertainty or mismatch is measured. A correct answer to one does not settle the others. A well-defined posterior may rely on a bad likelihood; a precise estimate may come from an unrepresentative sample; a lower training loss may fail to improve performance on new data.
Conditioning: The Logic of Updating on Evidence
For an event B with positive probability, P(A given B) = P(A and B) / P(B). Conditioning changes the reference population. Bayes’ rule expresses the same joint probability in two directions, allowing evidence about observations given a hypothesis to update uncertainty about the hypothesis given observations.
Suppose 1% of incoming messages are spam. A filter flags 90% of spam and 5% of legitimate messages. In a population of 10,000 messages with exactly these proportions, it flags 90 spam messages and 495 legitimate ones. Only 90 / 585 ≈ 15.4% of flagged messages are spam. The impressive detection rate does not determine the probability the user needs; the base rate and false-positive rate matter too.
A likelihood is a function of a model parameter with the observed data held fixed. Maximum likelihood chooses the parameter that maximizes it. Bayesian inference combines likelihood and prior to produce a posterior distribution. A maximum-a-posteriori estimate can coincide with maximum likelihood under a suitable flat prior on the parameter domain, but that point estimate is not the full posterior, and “flat” depends on the chosen parameterization.
Independence must be justified, not inferred from a lack of obvious connection. Repeated measurements from the same user may be correlated. In a graphical model, arrows and missing edges encode assumptions about a joint distribution; they are not automatically causal statements. The arithmetic can be flawless while the dependence assumptions are wrong.
Statistical Inference: Reading the Data
An expectation is an average weighted by probabilities. Suppose each of ten requests has a 10% chance of failure. Let Iᵢ be 1 when request i fails and 0 otherwise; this is an indicator variable. The total failure count is N = I₁ + … + I₁₀. Each indicator has expectation 0.1, so E[N] = 1. Adding expectations this way does not require independence.
The probability of at least one failure is a different quantity. With independent failures, it is 1 − 0.9¹⁰ ≈ 65.1%. If all ten requests instead share one failure event that occurs with probability 10%, it is 10%: either all fail or none do. Both models have the same expected failure count, but very different distributions. Estimating an average and estimating the risk of a particular event require different information about how outcomes depend on one another.
Separate an estimator’s variability from systematic error. For independent, identically distributed observations with finite variance σ², the sample mean has variance σ²/n. More samples reduce that sampling variance; they do not repair selection bias, mislabeled data, or a mismatch between the sampled and target populations.
The law of large numbers describes convergence of sample averages under its hypotheses. The central limit theorem describes the limiting distribution of appropriately centered and scaled averages under conditions such as independent, identically distributed observations with finite, nonzero variance. Neither says that an arbitrary small sample is approximately normal. A quantitative bound on how quickly the normal approximation improves requires additional conditions; it should not be confused with the standard error’s 1/√n scaling.
A 95% confidence-interval procedure is designed to cover the fixed parameter in 95% of repeated samples under its model, exactly or approximately depending on the method. A Bayesian 95% credible interval contains 95% of posterior probability under its prior and likelihood. These statements can yield similar numerical intervals while having different meanings.
A p-value is the probability, under a specified null model, of a test statistic at least as extreme as the observed one. It is not the probability that the null hypothesis is true. Statistical significance also does not measure effect size or practical importance. The ASA statement on p-values makes these distinctions explicit. Report the design, uncertainty, and magnitude of an effect, and account for data-dependent selection or multiple testing.
Causal claims require further support, such as a suitable randomized experiment or defensible identification assumptions. A/B testing illustrates why the unit of randomization matters: if treatment is assigned per user, treating every click as an independent experimental unit can exaggerate precision. Statistical reasoning begins with how the data were generated, not with choosing a formula after collection.
Information Theory: Measuring Uncertainty
For a discrete distribution, entropy H(P) = -Σ p(x) log₂ p(x) measures average uncertainty in bits. A fair coin has entropy one bit; an outcome known in advance has entropy zero. Entropy describes a distribution of possible outcomes, so it depends on what is already known and on the model used for the source.
Cross-entropy H(P,Q) = -Σ p(x) log₂ q(x) evaluates predictions made with Q when outcomes follow P. Suppose a coin is fair, but a model assigns probabilities (0.9, 0.1). Its cross-entropy is approximately 1.737 bits, compared with one bit when it predicts (0.5, 0.5). The model is penalized for repeatedly assigning too little probability to outcomes that occur often. Where the quantities are finite, H(P,Q) = H(P) + D_KL(P || Q); the excess in this example is approximately 0.737 bits.
KL divergence is nonnegative but generally asymmetric. Assigning zero model probability to an event with positive true probability makes D_KL(P || Q) infinite. Direction matters when choosing an approximation. If a posterior is difficult to compute, variational inference selects a simpler approximate distribution and commonly minimizes D_KL(q(z given x) || p(z given x)). This compares the approximate posterior with the target posterior. The original VAE paper relates this approximation to its training bound.
Mutual information answers a different question: how much does observing one variable reduce uncertainty about another, on average? For discrete variables, I(X;Y) = H(X) - H(X given Y). If X is a fair bit and Y is a perfect copy, knowing Y removes its one bit of uncertainty. If Y is an independent coin toss, it removes none. This gives a way to discuss whether an observation or feature supplies information about a target; information alone does not establish causation.
Negative log-likelihood connects many learning objectives to probability models. Squared error corresponds to a particular Gaussian-noise assumption; categorical cross-entropy corresponds to a categorical observation model. These are modeling choices, not universal descriptions of every useful loss. A predictive score can improve while calibration, subgroup performance, or usefulness for a decision remains poor. Ask what the objective measures and what it omits.
What Studying This Changes
The change is learning to ask what a number licenses you to conclude. An accuracy score requires a test population, a sampling process, and an error breakdown. A probability estimate needs calibration checks appropriate to its intended use. A difference between two systems needs an uncertainty assessment and a judgment about whether the magnitude matters.
Practice by simulating the spam-filter example, changing its base rate, and comparing predicted and observed proportions. Then analyze a small experiment by identifying the randomization unit, estimating an effect, and reporting uncertainty. These exercises connect probability arithmetic to statistical design—the part that cannot be recovered merely by running a more elaborate model afterward.
Resources
Books and Texts
Blitzstein and Hwang’s Introduction to Probability (2nd ed., 2019) is the best single entry to probability. Its distinguishing feature is pedagogical care: every concept is developed through examples, intuition is built alongside formalism, and conditional reasoning is emphasized throughout rather than treated as a special topic. The “story proofs” — intuitive arguments that precede formal proofs — train probabilistic intuition as much as technique. Blitzstein’s Harvard Stat 110 lectures (free on YouTube) are equally good and sometimes better; his live teaching reveals the reasoning behind the mathematics in ways that the written text occasionally compresses. Most learners should use the book and the lectures together.
Wasserman’s All of Statistics: A Concise Course in Statistical Inference (2004) is the most efficient route to statistical inference for mathematically prepared CS learners. It covers point estimation, confidence intervals, hypothesis testing, regression, nonparametric methods, and Bayesian inference in roughly 450 pages — far less than a conventional statistics text — without sacrificing rigor. The concision is the point: Wasserman covers what CS practitioners need without the applied examples that extend conventional statistics textbooks to 800 pages. Reading Blitzstein-Hwang then Wasserman is the most direct path from probability foundations to statistical inference for a CS practitioner.
For Bayesian inference with both depth and accessibility, McElreath’s Statistical Rethinking (2nd ed., 2020, with free lectures) is the most effective treatment available. McElreath teaches Bayesian reasoning as a coherent approach to learning from data, not as a technique to be applied after frequentist methods are learned. He insists on drawing generative models, deriving likelihoods from assumptions, and computing posteriors rather than looking up test statistics — habits that transfer directly to probabilistic ML. The book covers regression, multilevel models, and causal inference in a unified Bayesian framework. For practitioners who want to understand what probabilistic ML is doing at the inferential level, this is the right companion.
For information theory, Cover and Thomas’s Elements of Information Theory (2nd ed., 2006) is the standard rigorous treatment. It covers entropy, mutual information, channel capacity, source coding, and rate-distortion theory from first principles. MacKay’s Information Theory, Inference, and Learning Algorithms (free at inference.org.uk) covers similar content but unifies it with Bayesian inference and machine learning in a way that Cover-Thomas does not. MacKay is the book that reveals why cross-entropy loss and maximum likelihood estimation are the same thing, why KL divergence appears in variational inference, and why entropy regularization makes sense. For ML practitioners, MacKay is the more directly useful entry; for information theorists, Cover-Thomas is the more comprehensive reference.
For probabilistic ML specifically, Murphy’s Probabilistic Machine Learning: An Introduction (2022, free at probml.ai) is the comprehensive contemporary reference, covering probabilistic fundamentals alongside ML models in a unified framework. Bishop’s Pattern Recognition and Machine Learning (2006) remains the classical reference that defined the probabilistic ML approach. Both are treatment-at-depth references rather than first reads.
For CS-specific probability with algorithmic applications — randomized algorithms, hash functions, probabilistic data structures, concentration inequalities — Mitzenmacher and Upfal’s Probability and Computing (2nd ed., 2017) is the right text. It covers the probabilistic method, Markov chains, random graphs, and streaming algorithms from a CS perspective that neither Blitzstein-Hwang nor Wasserman addresses. A learner who wants both probabilistic foundations and algorithmic applications should read Blitzstein-Hwang then Mitzenmacher-Upfal.
For statistical learning theory — the mathematical analysis of when and why learning algorithms generalize — Shalev-Shwartz and Ben-David’s Understanding Machine Learning (free online) is the accessible entry, covering PAC learning, VC dimension, and the bias-complexity tradeoff with unusual clarity. Vershynin’s High-Dimensional Probability (free online) covers the concentration inequalities that are increasingly central to understanding modern overparameterized models.
For rigorous probability foundations, Williams’s Probability with Martingales is the standard accessible introduction to measure-theoretic probability. It develops the Lebesgue integral and martingale theory in about 250 pages with exceptional clarity — the right entry for learners who want to understand the mathematical foundations rather than just the computational tools.
| Book | Role | Type |
|---|---|---|
| Blitzstein & Hwang, Introduction to Probability (2nd ed.) | Canonical probability entry; conditional reasoning | Entry |
| Wasserman, All of Statistics (2004) | Concise statistical inference for CS | Depth |
| McElreath, Statistical Rethinking (2nd ed.) | Bayesian reasoning habits; generative models | Depth |
| Cover & Thomas, Elements of Information Theory (2nd ed.) | Information theory reference | Reference |
| MacKay, Information Theory, Inference, and Learning Algorithms (free) | Unified probability, inference, ML | Depth |
| Mitzenmacher & Upfal, Probability and Computing (2nd ed.) | Probability for algorithms and CS | Depth |
| Shalev-Shwartz & Ben-David, Understanding Machine Learning (free) | Statistical learning theory | Depth |
| Vershynin, High-Dimensional Probability (free) | Concentration inequalities; modern theory | Depth |
| Gelman et al., Bayesian Data Analysis (3rd ed.) | Comprehensive Bayesian reference | Reference |
| Williams, Probability with Martingales | Measure-theoretic foundations | Depth |
| Murphy, Probabilistic Machine Learning: An Introduction (free) | Comprehensive probabilistic ML | Auxiliary |
| Bishop, Pattern Recognition and Machine Learning | Classical probabilistic ML reference | Auxiliary |
| Koller & Friedman, Probabilistic Graphical Models | Graphical models reference | Auxiliary |
Courses and Lectures
Blitzstein’s Harvard Stat 110 (free on YouTube) is the most polished probability course freely available. Blitzstein’s teaching style builds probabilistic intuition through worked examples and careful attention to how people typically go wrong. Watching the lectures alongside the textbook is substantially more effective than either alone; his live reasoning is often more revealing than the written derivations.
MIT 6.041/6.431 (Probability, free on MIT OCW) is the MIT probability course, rigorous and comprehensive. It has a different emphasis from Stat 110 — more formal, less intuition-first — and serves as a good complement. Full lecture videos, problem sets, and exams are available.
McElreath’s Statistical Rethinking lectures (free on YouTube) accompany the textbook chapter by chapter. The lectures develop Bayesian reasoning habits through worked examples in R and Stan, making the abstract more concrete. The 2023 iteration of the lectures updates examples and includes contemporary perspectives on model checking and causal inference.
MIT 18.650 (Statistics for Applications, free on MIT OCW) covers statistical inference — estimation, hypothesis testing, regression, Bayesian methods — at graduate level. Complementary to Stat 110’s probability emphasis.
Berkeley Data 8 and its free text Computational and Inferential Thinking form a strong applied bridge from programming to statistics. The course uses Jupyter notebooks, real datasets, simulation, and inference, making it valuable for learners who need statistics to become something they can compute with rather than only something they can derive. It should not replace a probability/statistics spine, but it is one of the best applied companions.
StatQuest with Josh Starmer (YouTube, free, with optional paid materials) provides short visual explanations of probability, statistics, and machine-learning concepts: p-values, regression, logistic regression, Bayes, random forests, PCA, and related topics. Its main value is repair and intuition, not depth; use it when a concept remains opaque after a course or textbook explanation.
| Course | Platform | Type |
|---|---|---|
| Blitzstein, Stat 110 Probability (free) | YouTube / Harvard course site | Entry |
| MIT 6.041 Probability (free) | MIT OCW | Entry |
| McElreath, Statistical Rethinking lectures (free) | YouTube | Depth |
| MIT 18.650 Statistics for Applications (free) | MIT OCW | Depth |
| Berkeley Data 8 + Computational and Inferential Thinking (free) | Berkeley / Jupyter Book | Practice |
| StatQuest with Josh Starmer (free; optional paid materials) | YouTube / statquest.org | Auxiliary |
Practice, Tools, and Code
Seeing Theory (seeing-theory.brown.edu, free) is an interactive visual introduction to probability and statistics. It visualizes probability distributions, expected value, the law of large numbers, confidence intervals, and the central limit theorem interactively — the right supplement for concepts that are hard to build intuition for from text alone.
Implementing probabilistic concepts from scratch builds understanding that reading does not. Key projects: implement a maximum likelihood estimator for a Gaussian from data; implement Bayesian updating for a Beta-Binomial model and watch the posterior converge; implement importance sampling for a simple integral; implement a Metropolis-Hastings MCMC sampler for a distribution specified only up to a constant. Each takes a few hours and each builds a piece of the probabilistic machinery that ML frameworks use internally.
PyMC (Python) and Stan are the standard probabilistic programming systems for Bayesian computation, letting you specify probabilistic models declaratively and sample from posteriors automatically. Working through the PyMC tutorials reveals how MCMC connects to Bayesian inference in practice. scipy.stats provides standard distributions and statistical tests for computational practice.
For connecting statistical inference to ML, implement logistic regression by hand using gradient ascent on the log-likelihood, and verify that it matches the cross-entropy loss minimization that PyTorch would compute. This single exercise makes the MLE-cross-entropy-log-likelihood identity concrete and is more convincing than any written explanation.
| Resource | Platform | Type |
|---|---|---|
| Seeing Theory (free) | seeing-theory.brown.edu | Practice |
| PyMC tutorials (free) | docs.pymc.io | Practice |
| Stan documentation (free) | mc-stan.org | Practice |
| scipy.stats documentation (free) | scipy.org | Practice |
| Scratch implementations: MLE, Bayes updating, MCMC | Local | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Reversing a conditional probability | Detection rates do not determine the probability a flagged case is positive. | Write the joint probabilities or a frequency table, including base rates. |
| Treating maximum likelihood as full Bayesian inference | A maximizing parameter is not a posterior distribution. | Distinguish likelihood, prior, posterior, and the decision or summary extracted from them. |
| Assuming large samples remove every error | More data can make a biased estimate more precise. | Examine sampling, dependence, measurement, and population mismatch. |
| Reading a p-value as the chance a hypothesis is true | It is calculated under a model, using a specified statistic. | State the null model and report effect size and uncertainty alongside the test. |
| Inferring causation from predictive success | Prediction can exploit confounding or information unavailable at decision time. | Explain the design or assumptions supporting the causal claim. |
| Ignoring the direction of a divergence or meaning of a loss | Different objectives penalize different mismatches. | Name both distributions and connect the objective to the model and task. |
中文
概率在模型内描述不确定性；统计利用观测，估计、比较和检查模型，或关于总体的主张。两种活动相互联系，但区分它们很有用。给定一枚硬币出现正面的概率，概率论描述抛掷序列。给定一段序列，统计推断追问它支持关于硬币的哪些结论，以及“相互独立、概率固定”的模型是否合适。
不确定性有多种来源：主动引入的随机性、测量噪声、有限样本，以及知识不足。把概率解释为长期频率或信念程度，会引出不同的方法选择，但并不意味着概率具有两套不同的运算规则。概率数字应当与它的解释和前提一起出现。
离散例子需要代数和计数，连续分布引入积分，多变量模型则受益于线性代数。先研究小例子与模拟，再把它们连接到定义。模拟器可以在其编入的假设下核对推导，却不能证明这些假设符合我们关心的总体。
从赌桌到学习的数学
机会游戏推动了数学概率的发展，1654 年 Pascal 与 Fermat 的通信是一个熟悉的节点。但计数实践、保险、人口研究和测量也塑造了这一领域。大数定律在特定条件下，把重复观测与稳定频率联系起来；关于误差和估计的研究，则处理不完美的测量如何支持可靠结论。
二十世纪的统计发展出多种推断方式。Fisher 重视似然与显著性检验；Neyman 和 Pearson 发展控制错误率的检验；贝叶斯方法使用先验分布与条件化。它们提出的问题相关，却并不相同。Kolmogorov 在 1933 年建立的公理体系为概率提供了共同的数学基础，并不是裁定只有某种解释或推断方法才合法。
计算机改变了哪些量能够实际算出。模拟、重采样和 Markov 链 Monte Carlo，使许多难以精确求和或积分的模型能够使用。同时，统计问题旁边又出现了计算问题：链是否探索了相关分布，有效样本是否足够，估计是否稳定？增加计算量，不会自动克服混合不良或模型不合适的问题。
AI 的历史提供了另一个有用的启发。专家系统有时给规则附加非正式的置信分数，但一致地组合不确定性，不能只靠传递任意数值。概率图模型把依赖关系明确表示出来，并提供推断规则。统计学习也有模式识别和估计等来源，不能简单理解成神经网络整体取代逻辑。
信息论建立了另一条联系。Shannon 在 1948 年把不确定性与通信、编码连接起来；Kullback 和 Leibler 的 1951 年论文发展了以他们命名的散度。这些观念有助于解释压缩和学习目标，前提是先说清正在比较哪些分布。
条件化、估计与不确定性的度量
区分三个问题：观察到证据后什么发生变化，样本支持哪些推断，以及如何度量不确定性或不匹配程度。一个问题答对了，并不代表其他问题也解决了。定义清楚的后验可能依赖错误的似然模型；精确的估计可能来自缺乏代表性的样本；更低的训练损失可能没有改善新数据上的表现。
条件化：基于证据更新的逻辑
当事件 B 的概率为正时，P(A｜B) = P(A 且 B) / P(B)。条件化改变了参照总体。贝叶斯公式从两个方向表达同一个联合概率，使我们能够利用“假设成立时会出现什么观测”，更新“看到观测后对假设有多大把握”。
假设收到的邮件中有 1% 是垃圾邮件。过滤器能标记 90% 的垃圾邮件，同时把 5% 的正常邮件误标。若 10,000 封邮件恰好符合这些比例，它会标记 90 封垃圾邮件和 495 封正常邮件。因此，被标记邮件中只有 90 / 585 ≈ 15.4% 真是垃圾邮件。看起来很高的检出率，并不能独自确定用户需要的概率；基础发生率和误报率也很重要。
似然把观测数据固定，视为模型参数的函数。最大似然选择使它最大的参数。贝叶斯推断则结合似然与先验，得到后验分布。采用参数定义域上合适的平坦先验时，最大后验点估计可能与最大似然一致，但点估计并不等于完整后验，而且“平坦”依赖所选的参数化方式。
独立性需要依据，不能因为没有看见明显联系就默认成立。同一用户的多次测量可能相关。在图模型中，箭头和缺失的边编码了对联合分布的假设，并不会自动成为因果陈述。运算完全正确，依赖关系的假设仍可能错误。
统计推断：读懂数据
期望是按概率加权的平均值。假设十个请求各有 10% 的失败概率。让 Iᵢ 在第 i 个请求失败时取 1，否则取 0；这叫作指示变量。失败总数是 N = I₁ + … + I₁₀。每个指示变量的期望都是 0.1，所以 E[N] = 1。这样把期望相加，并不要求各次失败相互独立。
至少失败一次的概率是另一个量。如果各次失败独立，这个概率为 1 − 0.9¹⁰ ≈ 65.1%。如果十个请求共同受一个发生概率为 10% 的故障事件影响，它们要么全部失败，要么全部成功，至少失败一次的概率便是 10%。两个模型的期望失败数相同，分布却很不相同。估计平均值和估计某个事件的风险，对结果之间的依赖关系有不同的信息需求。
要区分估计量的变动与系统性错误。对于独立同分布、方差为有限值 σ² 的观测，样本均值的方差是 σ²/n。更多样本能减小这种抽样方差，却不能修复选择偏差、错误标签，或样本总体与目标总体之间的不匹配。
大数定律在其前提下描述样本均值的收敛。中心极限定理则在一定条件下，例如观测独立同分布且方差有限、非零时，描述经过适当中心化与缩放的均值的极限分布。两者都没有声称任意小样本已经近似正态。若要给出正态近似改善速度的定量界，还需要额外条件；不能把它与标准误差的 1/√n 缩放规律混为一谈。
一个 95% 置信区间构造程序，目标是在模型下重复抽样时，有 95% 的区间覆盖固定参数；视方法而定，这可能是精确或近似的保证。贝叶斯 95% 可信区间，则在所用先验与似然下包含 95% 的后验概率。两者可能产生相近的数值区间，含义却不同。
p 值是在指定的零假设模型下，检验统计量至少达到观测值那样极端的概率。它并不是零假设为真的概率。统计显著性也不衡量效应大小或实际重要性。ASA 关于 p 值的声明明确区分了这些含义。报告结果时，应同时说明研究设计、不确定性与效应大小，并处理依数据选择分析方式或多重检验带来的影响。
因果主张需要更多支持，例如合适的随机实验，或可以辩护的识别假设。A/B 测试说明了随机化单位为什么重要：如果按用户分配处理，把每次点击都当成独立实验单位，就可能夸大精度。统计推理应从数据怎样产生开始，而不是收集完数据后才选择公式。
信息论：度量不确定性
对于离散分布，熵 H(P) = -Σ p(x) log₂ p(x) 以比特衡量平均不确定性。公平硬币的熵是一比特；事先确定的结果，其熵为零。熵描述一组可能结果的分布，因此取决于已经知道什么，以及为信源采用了什么模型。
交叉熵 H(P,Q) = -Σ p(x) log₂ q(x) 评价这样一种预测：结果来自 P，预测却使用 Q。假设硬币公平，模型却分配概率 (0.9, 0.1)，交叉熵就约为 1.737 比特；正确预测 (0.5, 0.5) 时则是一比特。模型反复给经常出现的结果分配过低概率，因此受到惩罚。当这些量有限时，H(P,Q) = H(P) + D_KL(P || Q)；例子中多出的约 0.737 比特就是 KL 散度。
KL 散度非负，但一般不对称。当真实概率为正的事件被模型赋予零概率时，D_KL(P || Q) 会变成无穷大。选择近似时，方向很重要。如果后验难以计算，变分推断会选择较简单的近似分布，并通常最小化 D_KL(q(z｜x) || p(z｜x))。这里比较的是近似后验与目标后验。VAE 原始论文说明了这种近似与训练界的关系。
互信息回答另一个问题：平均而言，观察一个变量，会减少多少关于另一个变量的不确定性？对离散变量，I(X;Y) = H(X) - H(X｜Y)。如果 X 是公平的随机位，Y 完整复制了它，那么知道 Y 就消除了一比特不确定性。如果 Y 是另一次独立的抛硬币结果，就没有消除任何不确定性。这使我们能够讨论观测或特征是否提供了关于目标的信息；有信息本身并不能建立因果关系。
负对数似然把许多学习目标与概率模型连接起来。平方误差对应特定的高斯噪声假设；分类交叉熵对应分类观测模型。这些都是建模选择，不是所有有用损失函数的统一说明。预测分数改善时，概率校准、子群体表现，或对实际决策的帮助仍可能很差。应追问目标函数衡量了什么，又遗漏了什么。
学习这一部分会改变什么
变化在于学会追问：一个数字究竟允许我们得出什么结论？准确率需要测试总体、抽样过程和错误分类说明；概率估计需要与用途相符的校准检查；两个系统之间的差异，需要不确定性评估，以及差异大小是否重要的判断。
可以先模拟垃圾邮件过滤器的例子，改变基础发生率，比较预测与观测比例。再分析一个小实验，指出随机化单位，估计效应，并报告不确定性。这些练习把概率运算与统计设计联系起来；后者不能只靠事后运行更复杂的模型来补救。
资源
书籍与文本
Blitzstein 和 Hwang 的 Introduction to Probability（第 2 版，2019）是概率论最好的单本入口。它的独特之处在于教学上的细致：每个概念都通过例子发展，直觉与形式并行建立，而条件推理会贯穿始终，而不是被当作特殊主题处理。其中的 “story proofs”——先于形式证明的直觉论证——训练的不只是技巧，也是概率直觉。Blitzstein 的 Harvard Stat 110 讲座（YouTube 免费）同样优秀，有时甚至更好；他的现场教学会以文字偶尔压缩掉的方式展现数学背后的推理。多数学习者应把书和讲座结合使用。
Wasserman 的 All of Statistics: A Concise Course in Statistical Inference（2004）是数学准备较好的 CS 学习者进入统计推断的最高效路径。它在约 450 页中覆盖点估计、置信区间、假设检验、回归、非参数方法和贝叶斯推断——远短于传统统计教材——同时并未牺牲严谨性。简洁正是重点：Wasserman 覆盖了 CS 实践者所需内容，而没有用大量应用例子把传统统计教材扩展到 800 页。先读 Blitzstein-Hwang，再读 Wasserman，是 CS 实践者从概率基础进入统计推断的最直接路径。
对于兼具深度和可及性的贝叶斯推断，McElreath 的 Statistical Rethinking（第 2 版，2020，配套免费讲座）是目前最有效的处理。McElreath 把贝叶斯推理作为一种从数据中学习的连贯方式来教授，而不是把它当作在学完频率主义方法后再补充的技术。他坚持画生成模型，从假设推导似然，并计算后验，而不是查检验统计量——这些习惯会直接迁移到概率机器学习中。这本书在一个统一的贝叶斯框架中覆盖回归、多层模型和因果推断。对于想在推断层面理解概率 ML 在做什么的实践者来说，这是正确伴读材料。
对于信息论，Cover 和 Thomas 的 Elements of Information Theory（第 2 版，2006）是标准严谨处理。它从第一原则出发覆盖熵、互信息、信道容量、源编码和率失真理论。MacKay 的 Information Theory, Inference, and Learning Algorithms（可在 inference.org.uk 免费获取）覆盖类似内容，但以 Cover-Thomas 所没有的方式，把它与贝叶斯推断和机器学习统一起来。MacKay 这本书会揭示为什么交叉熵损失和最大似然估计是同一件事，为什么 KL 散度出现在变分推断中，以及为什么熵正则化是合理的。对 ML 实践者来说，MacKay 是更直接有用的入口；对信息论研究者来说，Cover-Thomas 是更全面的参考。
对于概率机器学习本身，Murphy 的 Probabilistic Machine Learning: An Introduction（2022，可在 probml.ai 免费获取）是当代综合参考，在统一框架中同时覆盖概率基础和 ML 模型。Bishop 的 Pattern Recognition and Machine Learning（2006）仍然是定义了概率 ML 方法的经典参考。二者都适合作为深入处理的参考，而不是第一本书。
对于具有算法应用的CS 专门概率——随机算法、哈希函数、概率数据结构、集中不等式——Mitzenmacher 和 Upfal 的 Probability and Computing（第 2 版，2017）是正确教材。它从 CS 视角覆盖概率方法、Markov 链、随机图和流式算法，而这些内容不是 Blitzstein-Hwang 或 Wasserman 的重点。想同时获得概率基础和算法应用的学习者，应先读 Blitzstein-Hwang，再读 Mitzenmacher-Upfal。
对于统计学习理论——对学习算法何时以及为何能够泛化的数学分析——Shalev-Shwartz 和 Ben-David 的 Understanding Machine Learning（免费在线）是可进入的入口，以异常清晰的方式覆盖 PAC 学习、VC 维和偏差—复杂度权衡。Vershynin 的 High-Dimensional Probability（免费在线）覆盖集中不等式，而这在理解现代过参数化模型时越来越关键。
对于严谨概率基础，Williams 的 Probability with Martingales 是测度论概率的标准可进入导论。它在约 250 页中发展 Lebesgue 积分和鞅理论，并且极其清晰——对于想理解数学基础，而不只是掌握计算工具的学习者，这是合适入口。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Blitzstein & Hwang, Introduction to Probability（第 2 版） | 经典概率入口；条件推理 | 入门 |
| Wasserman, All of Statistics（2004） | 面向 CS 的简明统计推断 | 深入 |
| McElreath, Statistical Rethinking（第 2 版） | 贝叶斯推理习惯；生成模型 | 深入 |
| Cover & Thomas, Elements of Information Theory（第 2 版） | 信息论参考 | 参考 |
| MacKay, Information Theory, Inference, and Learning Algorithms（免费） | 统一概率、推断与 ML | 深入 |
| Mitzenmacher & Upfal, Probability and Computing（第 2 版） | 面向算法和 CS 的概率 | 深入 |
| Shalev-Shwartz & Ben-David, Understanding Machine Learning（免费） | 统计学习理论 | 深入 |
| Vershynin, High-Dimensional Probability（免费） | 集中不等式；现代理论 | 深入 |
| Gelman et al., Bayesian Data Analysis（第 3 版） | 综合贝叶斯参考 | 参考 |
| Williams, Probability with Martingales | 测度论基础 | 深入 |
| Murphy, Probabilistic Machine Learning: An Introduction（免费） | 综合概率 ML | 辅助 |
| Bishop, Pattern Recognition and Machine Learning | 经典概率 ML 参考 | 辅助 |
| Koller & Friedman, Probabilistic Graphical Models | 图模型参考 | 辅助 |
课程与讲座
Blitzstein 的 Harvard Stat 110（YouTube 免费）是目前可免费获取的最精致概率课程。Blitzstein 的教学风格通过完整例题和对常见错误的细致关注来建立概率直觉。把讲座与教材一起观看，效果显著强于单独使用其中任何一个；他的现场推理往往比书面推导更有揭示性。
MIT 6.041/6.431（Probability，MIT OCW 免费）是 MIT 的概率课程，严谨而全面。它与 Stat 110 的侧重点不同——更形式化，不那么以直觉优先——可以作为很好的补充。完整讲课视频、习题和考试都可获取。
McElreath 的 Statistical Rethinking 讲座（YouTube 免费）逐章配合教材。讲座通过 R 和 Stan 中的完整例子发展贝叶斯推理习惯，使抽象概念更具体。2023 年版本的讲座更新了例子，并包含关于模型检查和因果推断的当代视角。
MIT 18.650（Statistics for Applications，MIT OCW 免费）以研究生水平覆盖统计推断——估计、假设检验、回归、贝叶斯方法。它与 Stat 110 的概率重点互补。
Berkeley Data 8 及其免费教材 Computational and Inferential Thinking 构成了从编程到统计的强应用桥梁。这门课使用 Jupyter notebook、真实数据集、仿真和推断，非常适合那些需要把统计变成可以计算之物，而不仅是可以推导之物的学习者。它不应替代概率/统计主干，但它是最好的应用伴读之一。
StatQuest with Josh Starmer（YouTube 免费，附可选付费材料）为概率、统计和机器学习概念提供短小的视觉解释：p 值、回归、逻辑回归、Bayes、随机森林、PCA 及相关主题。它的主要价值是修复理解和建立直觉，而不是深度；当一个概念在课程或教材解释之后仍然不透明时，可以使用它。
| 课程 | 平台 | 类型 |
|---|---|---|
| Blitzstein, Stat 110 Probability（免费） | YouTube / Harvard 课程网站 | 入门 |
| MIT 6.041 Probability（免费） | MIT OCW | 入门 |
| McElreath, Statistical Rethinking lectures（免费） | YouTube | 深入 |
| MIT 18.650 Statistics for Applications（免费） | MIT OCW | 深入 |
| Berkeley Data 8 + Computational and Inferential Thinking（免费） | Berkeley / Jupyter Book | 实践 |
| StatQuest with Josh Starmer（免费；付费材料可选） | YouTube / statquest.org | 辅助 |
实践、工具与代码
Seeing Theory（seeing-theory.brown.edu，免费）是一个交互式概率和统计视觉入门。它交互式展示概率分布、期望值、大数定律、置信区间和中心极限定理——对于那些很难只靠文本建立直觉的概念，它是合适补充。
从零实现概率概念，会建立阅读无法带来的理解。关键项目包括：从数据中实现 Gaussian 的最大似然估计；实现 Beta-Binomial 模型的贝叶斯更新，并观察后验如何收敛；为一个简单积分实现重要性采样；为一个只给定到归一化常数的分布实现 Metropolis-Hastings MCMC 采样器。每个项目需要几小时，每个项目都会建立机器学习框架内部使用的一部分概率机制。
PyMC（Python）和 Stan 是贝叶斯计算的标准概率编程系统，允许你以声明式方式指定概率模型，并自动从后验中采样。学习 PyMC 教程，会揭示 MCMC 如何在实践中连接到贝叶斯推断。scipy.stats 提供标准分布和统计检验，是计算实践中的常用工具。
要把统计推断连接到 ML，可以手写实现逻辑回归：对对数似然做梯度上升，并验证它与 PyTorch 所计算的交叉熵损失最小化相匹配。这个单一练习会让 MLE—交叉熵—对数似然之间的恒等式变得具体，比任何文字解释都更有说服力。
| 资源 | 平台 | 类型 |
|---|---|---|
| Seeing Theory（免费） | seeing-theory.brown.edu | 实践 |
| PyMC tutorials（免费） | docs.pymc.io | 实践 |
| Stan documentation（免费） | mc-stan.org | 实践 |
| scipy.stats documentation（免费） | scipy.org | 实践 |
| 从零实现：MLE、Bayes 更新、MCMC | 本地 | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 颠倒条件概率 | 检出率不能决定被标记对象为阳性的概率。 | 写出联合概率或频数表，包括基础发生率。 |
| 把最大似然当成完整贝叶斯推断 | 使似然最大的参数不是后验分布。 | 区分似然、先验、后验，以及从中提取的决策或摘要。 |
| 认为大样本能消除所有错误 | 更多数据可能只是让有偏估计更加精确。 | 检查抽样、依赖、测量和总体不匹配。 |
| 把 p 值读成假设为真的概率 | 它是在模型下，对指定统计量计算的量。 | 说明零假设模型，同时报告效应大小与不确定性。 |
| 从预测成功推断因果 | 预测可能利用混杂，或利用决策时尚不可得的信息。 | 说明支持因果主张的设计或假设。 |
| 忽略散度方向或损失含义 | 不同目标惩罚不同的不匹配。 | 说清两个分布，并把目标函数与模型和任务连接起来。 |