English
Probability is the mathematics of uncertainty. Where calculus describes how quantities change continuously and linear algebra describes how quantities transform linearly, probability describes how quantities behave when they are random — when the outcome cannot be predicted with certainty, only characterized in terms of how likely different outcomes are. A probability distribution does not say what will happen; it says how often each outcome would appear if the situation were repeated many times, or equivalently, how confident we should be in each possibility given what we know.
Statistics is the discipline of drawing conclusions from data under uncertainty. A probability model describes a data-generating process; statistics runs the inference in reverse: given data, what can we conclude about the process that generated it? This includes estimating unknown parameters, testing whether observed patterns are likely to be real rather than accidental, and quantifying the uncertainty in our conclusions. The two subjects are inseparable: statistics requires probability to define what “likely” and “confident” mean, and probability alone tells you nothing until data enters.
Information theory is probability applied to the measurement of uncertainty itself. It asks: how much uncertainty does a probability distribution represent? How much does observing one variable tell you about another? How many bits are needed, on average, to encode a message drawn from a given distribution? The central quantities — entropy, mutual information, KL divergence — appear throughout machine learning as loss functions, regularizers, and evaluation metrics, often without being labeled as information theory.
These three subjects are taught separately in most curricula and unified in practice. Machine learning is built on all three simultaneously: a language model defines a probability distribution, is trained by minimizing a statistical loss (cross-entropy), and is evaluated by information-theoretic metrics (perplexity). Understanding where each concept comes from and what it actually says prevents the confusion that arises when they are encountered in isolation.
Prerequisites: Calculus (§2.4) is needed for continuous probability, particularly probability density functions and expectations as integrals. Discrete probability (§2.2) is the finite special case. This section is prerequisite to Chapter 5 (Artificial Intelligence) and parts of §4.5.
From Gambling Tables to the Mathematics of Learning
The formal study of probability began with gambling. In the summer of 1654, Blaise Pascal and Pierre de Fermat exchanged letters about a problem posed by a gambler: if a game of chance is interrupted before it finishes, how should the stakes be divided? The question required reasoning about events that had not yet happened — assigning numbers to possibilities. The correspondence between Pascal and Fermat established the foundational rules: how to count equally likely outcomes, how to compute the probability of compound events, how to think about expected value as a guide to fair betting. They had no formal framework; they were solving specific problems. But the problems forced them to develop the tools, and the tools proved general.
Through the seventeenth and eighteenth centuries, probability spread outward from gambling into natural philosophy, demography, and error theory. Jacob Bernoulli proved the first limit theorem in 1713: if you repeat an experiment many times, the observed frequency converges to the true probability. This law of large numbers said something deep — that randomness in individual outcomes is compatible with regularity in aggregate behavior, and that probability is empirically meaningful, not just a bookkeeping convention. Abraham de Moivre discovered in 1733 that the sum of many independent random events tends toward a bell-shaped curve — the precursor to the central limit theorem that Pierre-Simon Laplace would prove in full generality in 1812. Carl Friedrich Gauss used the bell curve in 1809 to justify least-squares as the optimal method for combining astronomical measurements with random errors, connecting probability to the practice of science.
The nineteenth century added statistics. Adolphe Quetelet applied probability to social data in the 1830s and 1840s, showing that crime rates, birth rates, and physical measurements of populations followed stable statistical patterns — launching the quantitative social sciences. Francis Galton invented the concept of regression in the 1880s, discovering that the heights of children of tall parents tend toward the population average, not the parental extreme. Karl Pearson systematized Galton’s ideas and invented the chi-squared test and the correlation coefficient, giving statistics its first formal hypothesis-testing tools. By 1900, probability and statistics had become a unified discipline with both mathematical foundations and empirical applications — but the foundations were still informal. What exactly was a probability? A frequency in repeated trials? A degree of belief? The answer had been left implicit.
The twentieth century provided two incompatible answers. Andrey Kolmogorov’s 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung — “Foundations of the Theory of Probability” — put probability on the same axiomatic basis as any other branch of mathematics. The axioms are simple: probabilities are non-negative, the probability of the entire sample space is one, and the probability of a union of mutually exclusive events equals the sum of their probabilities. From these three axioms, all of probability theory follows rigorously. Kolmogorov’s axioms said nothing about what probability meant, only how it behaved. That question sparked a philosophical battle that has not been resolved.
Ronald Fisher, the most influential statistician of the twentieth century, championed the frequentist interpretation: a probability is the limiting frequency of an event in an infinite sequence of independent repetitions. This made probability objective and empirically testable, but it made some probability statements awkward — the probability that a specific unknown parameter lies in a specific interval, for instance, is either zero or one in the frequentist view, because the parameter is fixed even if unknown. Fisher developed maximum likelihood estimation, p-values, and analysis of variance in this framework, creating the methodology that dominated empirical science for most of the twentieth century.
Bruno de Finetti and Frank Ramsey, working independently in the late 1920s and 1930s, defended the Bayesian interpretation: a probability is a degree of belief, coherently updated by evidence using Bayes’ theorem. On this view, it is perfectly meaningful to assign a probability to a specific parameter — it expresses how confident we are about its value given the data we have. Harold Jeffreys developed objective Bayesian methods in the 1930s and 1940s, attempting to define prior distributions that expressed ignorance rather than arbitrary belief. The Bayesian-frequentist divide became the defining intellectual fault line of twentieth-century statistics.
The fault line was partly dissolved, not by philosophical argument but by computation. MCMC — Markov chain Monte Carlo — made Bayesian inference computationally tractable. The Metropolis algorithm was developed in 1953 by a team at Los Alamos including Nicholas Metropolis and Edward Teller, initially for physics simulations; Gelfand and Smith’s 1990 paper showed how to apply it to arbitrary Bayesian posterior distributions. Where Bayesian inference had previously been limited to problems with conjugate priors (where the posterior has the same form as the prior), MCMC made it possible to sample from posteriors of arbitrary complexity. Combined with increasing computing power, this opened the door to Bayesian approaches across statistics, machine learning, and scientific computing through the 1990s and 2000s.
Information theory arrived from a different direction entirely. Claude Shannon’s 1948 paper “A Mathematical Theory of Communication” asked: how much information is contained in a message? Shannon defined the entropy of a probability distribution as H = -Σ p(x) log p(x) — a measure of the distribution’s average uncertainty. High entropy means many equally likely outcomes; low entropy means most of the probability is concentrated. Shannon’s fundamental theorem proved that entropy is the minimum average number of bits needed to encode messages from a given distribution, connecting the abstract measure of uncertainty to the concrete problem of efficient communication. KL divergence — the information gained when one updates from one distribution to another — appeared in the same paper as a measure of how different two distributions are. Cross-entropy appeared as the average code length when you encode messages from distribution p using a code optimized for distribution q: minimizing cross-entropy is equivalent to maximizing the likelihood of the data under the model, making cross-entropy loss and maximum likelihood estimation two names for the same objective.
Shannon’s information theory and Kolmogorov’s probability theory were developed independently, but their combination became the theoretical foundation of machine learning. The training of a language model is maximum likelihood estimation, which is minimization of cross-entropy, which is minimization of KL divergence between the model distribution and the data distribution. The generalization of a classifier is bounded by its Rademacher complexity, a probabilistic quantity. The representations learned by neural networks can be analyzed through the information bottleneck — a mutual information framework. Statistical learning theory, which asks when and why machine learning algorithms generalize from training data to new data, uses probability theory throughout and its central results (VC dimension, PAC bounds) are proved using combinatorial probability. The contemporary discipline of machine learning is, at its mathematical core, a branch of probability and statistics augmented by computation.
Conditioning, Estimation, and the Measurement of Uncertainty
Conditioning: The Logic of Updating on Evidence
| Conditional probability is the central concept of probabilistic reasoning. P(A | B) — “the probability of A given B” — captures how the probability of one event changes when another event is known to have occurred. Its definition is simple: P(A | B) = P(A ∩ B)/P(B). But the concept is far more than a formula. It is the correct answer to the question “what does knowing B tell me about A?” and it governs all Bayesian reasoning. |
| Bayes’ theorem follows from this definition by simple algebra: P(θ | data) = P(data | θ)P(θ)/P(data). The likelihood P(data | θ) measures how probable the observed data would be if the parameter were θ. The prior P(θ) captures initial beliefs about θ before the data. The posterior P(θ | data) is the updated belief after seeing the data. The denominator P(data) is a normalizing constant that ensures the posterior integrates to one. This is the complete logic of Bayesian learning: beliefs are updated by evidence, with the likelihood function mediating between prior and posterior. |
| Maximum likelihood estimation (MLE) is the special case of Bayesian estimation with a uniform prior: among all possible parameter values, choose the one that makes the observed data most probable. This is equivalent to choosing the parameter that maximizes P(data | θ), which is the likelihood. The remarkable fact is that training a neural network by minimizing cross-entropy loss is exactly MLE under a categorical distribution: the model learns the parameters that make the training data most probable under its probability model. This connection — cross-entropy loss equals negative log-likelihood equals MLE objective — is not an analogy; it is an identity. |
| The independence assumption simplifies conditioning dramatically. Two events A and B are independent if P(A | B) = P(A): knowing B tells you nothing about A. Independence allows joint probabilities to factor: P(A ∩ B) = P(A)P(B). In practice, true independence is rare, but conditional independence — A is independent of B given C — is the structural assumption underlying probabilistic graphical models, naive Bayes classifiers, and Markov chains. The chain rule of probability expresses any joint distribution as a product of conditionals: P(X₁, …, Xₙ) = P(X₁)P(X₂ | X₁)…P(Xₙ | X₁,…,Xₙ₋₁). This is also the autoregressive factorization that language models use: P(word₁, word₂, …) = P(word₁)·P(word₂ | word₁)·P(word₃ | word₁,word₂)···. The architecture that produces GPT is, in probabilistic terms, an autoregressive model over this factorization. |
The central limit theorem is the great limit theorem connecting conditioning to aggregate behavior. When you average many independent samples from a distribution with finite mean and variance, the distribution of that average approaches a Gaussian, regardless of the original distribution. The rate of convergence is O(1/√n): averaging n samples reduces the standard deviation of the estimate by a factor of √n. This theorem is why frequentist confidence intervals have the form they do, why least squares is optimal under Gaussian noise, and why the law of large numbers holds in the sense it does — it is the mathematical foundation of statistical inference from samples.
Statistical Inference: Reading the Data
Statistical inference is the discipline of drawing defensible conclusions from finite data. It operates in the shadow of the central limit theorem: whatever the underlying distribution, large samples behave predictably, and this predictability is the source of statistical power.
Estimation asks: given data, what is the best guess for an unknown quantity? Point estimation produces a single number; interval estimation produces a range of plausible values. The frequentist confidence interval, often misunderstood, is a statement about the procedure rather than the specific interval: if you constructed confidence intervals by this method on many independent datasets, 95% of them would contain the true parameter. It is not a statement that the true parameter is in this particular interval with 95% probability — in the frequentist view, the true parameter is fixed, and the specific interval either contains it or does not. The Bayesian credible interval, by contrast, is a statement about probability: given the data and the prior, there is a 95% probability that the parameter lies in this range. The two intervals are often numerically similar but logically distinct.
Hypothesis testing asks: is the observed effect real, or is it plausible under the null hypothesis of no effect? Fisher’s p-value is the probability of observing data at least as extreme as what was seen, assuming the null hypothesis is true. A small p-value is evidence against the null hypothesis; it is not evidence for the alternative, nor is it the probability that the null is false. The Neyman-Pearson framework formalizes this as a decision rule: set a significance level α, reject the null if the p-value falls below α, and accept that this procedure makes false rejections (Type I errors) at rate α. The widespread misunderstanding of what p-values actually say has contributed significantly to the replication crisis that emerged after 2010, when large-scale replication attempts showed that many published findings in psychology, medicine, and social science did not reproduce.
Confounding and causation are the deepest difficulty in statistics. A correlation between two variables — cigarette smoking and lung cancer, say — might reflect a causal effect, or it might be explained by a confounding variable that causes both. Observational data, no matter how large the sample, cannot by itself distinguish correlation from causation; only experimental design (randomization to treatment and control) or causal assumptions encoded in a graph can make causal conclusions defensible. Pearl’s do-calculus and the potential outcomes framework of Rubin and Holland provide the formal tools for causal inference. This matters for ML because models trained on observational data learn correlations; when the data-generating process changes, the correlations can break while the causal structure remains stable.
Information Theory: Measuring Uncertainty
Entropy quantifies how uncertain a probability distribution is. For a discrete distribution, H(X) = -Σ p(x) log₂ p(x) bits. A deterministic distribution — one outcome with probability one — has zero entropy: there is no uncertainty. A uniform distribution over n outcomes has log₂ n bits of entropy: maximum uncertainty. Entropy measures the average surprise of observing an outcome: outcomes that happen rarely are more surprising than outcomes that happen often, and Shannon defined surprise as -log p(x).
| KL divergence D_KL(P | Q) = Σ p(x) log(p(x)/q(x)) measures how different distribution Q is from distribution P. It is not symmetric — D_KL(P | Q) ≠ D_KL(Q | P) in general — and this asymmetry is meaningful. The forward KL (P | Q) is small when Q assigns high probability wherever P assigns high probability; minimizing forward KL produces mode-covering behavior. The reverse KL (Q | P) is small when Q is concentrated in regions where P is large; minimizing reverse KL produces mode-seeking behavior. Variational autoencoders minimize a bound on the forward KL. Many other generative model training objectives can be understood as choices between forward and reverse KL. |
| Cross-entropy H(P, Q) = -Σ p(x) log q(x) measures the average code length when messages drawn from P are encoded using a code optimized for Q. It decomposes as H(P, Q) = H(P) + D_KL(P | Q): the cross-entropy equals the entropy of the true distribution plus the KL divergence between the true and model distributions. When P is the data distribution and Q is the model, minimizing cross-entropy is equivalent to minimizing the KL divergence from the data distribution to the model — that is, to making the model as close as possible to the true data distribution. This is the objective of every language model, image generative model, and density estimator. |
| Mutual information I(X; Y) = H(X) - H(X | Y) measures how much knowing Y reduces uncertainty about X. It is symmetric — I(X;Y) = I(Y;X) — and equals zero exactly when X and Y are independent. Mutual information appears in feature selection (selecting features with high mutual information to the target), in representation learning (the information bottleneck principle), and in the analysis of attention patterns in neural networks. Unlike correlation, mutual information captures nonlinear dependencies. |
What Studying This Changes
Probability and statistics change how a practitioner interprets evidence and uncertainty, not just how they compute with it.
The most consequential change is epistemic: the discipline of conditional reasoning. Probabilistically trained practitioners automatically ask, when evaluating a claim, what the prior probability is, what the likelihood ratio is, and what evidence would shift their belief. This Bayesian epistemology is not confined to formal calculations — it is a thinking style that distinguishes careful empirical reasoning from motivated reasoning. The replication crisis in science is largely a failure of this thinking style: effects that looked statistically significant on small samples and against weak null hypotheses were published as established findings; when the prior probability of the specific effect was low and the sample sizes were small, a significant p-value was much weaker evidence than it appeared.
The second change is the ability to design and evaluate experiments. Causal claims require causal structure — either randomization or explicit causal assumptions — not just statistical associations. A practitioner who has absorbed the difference between observational and experimental data, who understands confounding and selection bias, who knows what randomization buys, can evaluate whether a claimed result is actually supported by its evidence. This is a prerequisite for meaningful empirical work in any field that collects data, which is most of CS.
The third change is literacy in the probabilistic vocabulary of machine learning. Loss functions are log-likelihoods. Regularization is a prior. The variational autoencoder’s training objective is a lower bound on the log-likelihood. Score matching is equivalent to minimizing Fisher divergence. Contrastive learning optimizes a ratio of likelihoods. These connections are not metaphors; they are identities, and understanding them changes which architectural and algorithmic choices are interpretable and which are ad hoc.
The fourth change is calibrated uncertainty. A practitioner without statistical training tends to treat model outputs as facts. A practitioner with statistical training asks: what is the confidence interval on this estimate? Is this model calibrated — do its confidence scores match empirical frequencies? What sample size would be needed to detect this effect reliably? These questions have definite answers, and the ability to ask and answer them is what separates scientific ML practice from engineering intuition.
Resources
Books and Texts
Blitzstein and Hwang’s Introduction to Probability (2nd ed., 2019) is the best single entry to probability. Its distinguishing feature is pedagogical care: every concept is developed through examples, intuition is built alongside formalism, and conditional reasoning is emphasized throughout rather than treated as a special topic. The “story proofs” — intuitive arguments that precede formal proofs — train probabilistic intuition as much as technique. Blitzstein’s Harvard Stat 110 lectures (free on YouTube) are equally good and sometimes better; his live teaching reveals the reasoning behind the mathematics in ways that the written text occasionally compresses. Most learners should use the book and the lectures together.
Wasserman’s All of Statistics: A Concise Course in Statistical Inference (2004) is the most efficient route to statistical inference for mathematically prepared CS learners. It covers point estimation, confidence intervals, hypothesis testing, regression, nonparametric methods, and Bayesian inference in roughly 450 pages — far less than a conventional statistics text — without sacrificing rigor. The concision is the point: Wasserman covers what CS practitioners need without the applied examples that extend conventional statistics textbooks to 800 pages. Reading Blitzstein-Hwang then Wasserman is the most direct path from probability foundations to statistical inference for a CS practitioner.
For Bayesian inference with both depth and accessibility, McElreath’s Statistical Rethinking (2nd ed., 2020, with free lectures) is the most effective treatment available. McElreath teaches Bayesian reasoning as a coherent approach to learning from data, not as a technique to be applied after frequentist methods are learned. He insists on drawing generative models, deriving likelihoods from assumptions, and computing posteriors rather than looking up test statistics — habits that transfer directly to probabilistic ML. The book covers regression, multilevel models, and causal inference in a unified Bayesian framework. For practitioners who want to understand what probabilistic ML is doing at the inferential level, this is the right companion.
For information theory, Cover and Thomas’s Elements of Information Theory (2nd ed., 2006) is the standard rigorous treatment. It covers entropy, mutual information, channel capacity, source coding, and rate-distortion theory from first principles. MacKay’s Information Theory, Inference, and Learning Algorithms (free at inference.org.uk) covers similar content but unifies it with Bayesian inference and machine learning in a way that Cover-Thomas does not. MacKay is the book that reveals why cross-entropy loss and maximum likelihood estimation are the same thing, why KL divergence appears in variational inference, and why entropy regularization makes sense. For ML practitioners, MacKay is the more directly useful entry; for information theorists, Cover-Thomas is the more comprehensive reference.
For probabilistic ML specifically, Murphy’s Probabilistic Machine Learning: An Introduction (2022, free at probml.ai) is the comprehensive contemporary reference, covering probabilistic fundamentals alongside ML models in a unified framework. Bishop’s Pattern Recognition and Machine Learning (2006) remains the classical reference that defined the probabilistic ML approach. Both are treatment-at-depth references rather than first reads.
For CS-specific probability with algorithmic applications — randomized algorithms, hash functions, probabilistic data structures, concentration inequalities — Mitzenmacher and Upfal’s Probability and Computing (2nd ed., 2017) is the right text. It covers the probabilistic method, Markov chains, random graphs, and streaming algorithms from a CS perspective that neither Blitzstein-Hwang nor Wasserman addresses. A learner who wants both probabilistic foundations and algorithmic applications should read Blitzstein-Hwang then Mitzenmacher-Upfal.
For statistical learning theory — the mathematical analysis of when and why learning algorithms generalize — Shalev-Shwartz and Ben-David’s Understanding Machine Learning (free online) is the accessible entry, covering PAC learning, VC dimension, and the bias-complexity tradeoff with unusual clarity. Vershynin’s High-Dimensional Probability (free online) covers the concentration inequalities that are increasingly central to understanding modern overparameterized models.
For rigorous probability foundations, Williams’s Probability with Martingales is the standard accessible introduction to measure-theoretic probability. It develops the Lebesgue integral and martingale theory in about 250 pages with exceptional clarity — the right entry for learners who want to understand the mathematical foundations rather than just the computational tools.
| Book | Role | Type |
|---|---|---|
| Blitzstein & Hwang, Introduction to Probability (2nd ed.) | Canonical probability entry; conditional reasoning | Entry |
| Wasserman, All of Statistics (2004) | Concise statistical inference for CS | Depth |
| McElreath, Statistical Rethinking (2nd ed.) | Bayesian reasoning habits; generative models | Depth |
| Cover & Thomas, Elements of Information Theory (2nd ed.) | Information theory reference | Reference |
| MacKay, Information Theory, Inference, and Learning Algorithms (free) | Unified probability, inference, ML | Depth |
| Mitzenmacher & Upfal, Probability and Computing (2nd ed.) | Probability for algorithms and CS | Depth |
| Shalev-Shwartz & Ben-David, Understanding Machine Learning (free) | Statistical learning theory | Depth |
| Vershynin, High-Dimensional Probability (free) | Concentration inequalities; modern theory | Depth |
| Gelman et al., Bayesian Data Analysis (3rd ed.) | Comprehensive Bayesian reference | Reference |
| Williams, Probability with Martingales | Measure-theoretic foundations | Depth |
| Murphy, Probabilistic Machine Learning: An Introduction (free) | Comprehensive probabilistic ML | Auxiliary |
| Bishop, Pattern Recognition and Machine Learning | Classical probabilistic ML reference | Auxiliary |
| Koller & Friedman, Probabilistic Graphical Models | Graphical models reference | Auxiliary |
Courses and Lectures
Blitzstein’s Harvard Stat 110 (free on YouTube) is the most polished probability course freely available. Blitzstein’s teaching style builds probabilistic intuition through worked examples and careful attention to how people typically go wrong. Watching the lectures alongside the textbook is substantially more effective than either alone; his live reasoning is often more revealing than the written derivations.
MIT 6.041/6.431 (Probability, free on MIT OCW) is the MIT probability course, rigorous and comprehensive. It has a different emphasis from Stat 110 — more formal, less intuition-first — and serves as a good complement. Full lecture videos, problem sets, and exams are available.
McElreath’s Statistical Rethinking lectures (free on YouTube) accompany the textbook chapter by chapter. The lectures develop Bayesian reasoning habits through worked examples in R and Stan, making the abstract more concrete. The 2023 iteration of the lectures updates examples and includes contemporary perspectives on model checking and causal inference.
MIT 18.650 (Statistics for Applications, free on MIT OCW) covers statistical inference — estimation, hypothesis testing, regression, Bayesian methods — at graduate level. Complementary to Stat 110’s probability emphasis.
Berkeley Data 8 and its free text Computational and Inferential Thinking form a strong applied bridge from programming to statistics. The course uses Jupyter notebooks, real datasets, simulation, and inference, making it valuable for learners who need statistics to become something they can compute with rather than only something they can derive. It should not replace a probability/statistics spine, but it is one of the best applied companions.
StatQuest with Josh Starmer (YouTube, free, with optional paid materials) provides short visual explanations of probability, statistics, and machine-learning concepts: p-values, regression, logistic regression, Bayes, random forests, PCA, and related topics. Its main value is repair and intuition, not depth; use it when a concept remains opaque after a course or textbook explanation.
| Course | Platform | Type |
|---|---|---|
| Blitzstein, Stat 110 Probability (free) | YouTube / Harvard course site | Entry |
| MIT 6.041 Probability (free) | MIT OCW | Entry |
| McElreath, Statistical Rethinking lectures (free) | YouTube | Depth |
| MIT 18.650 Statistics for Applications (free) | MIT OCW | Depth |
| Berkeley Data 8 + Computational and Inferential Thinking (free) | Berkeley / Jupyter Book | Practice |
| StatQuest with Josh Starmer (free; optional paid materials) | YouTube / statquest.org | Auxiliary |
Practice, Tools, and Code
Seeing Theory (seeing-theory.brown.edu, free) is an interactive visual introduction to probability and statistics. It visualizes probability distributions, expected value, the law of large numbers, confidence intervals, and the central limit theorem interactively — the right supplement for concepts that are hard to build intuition for from text alone.
Implementing probabilistic concepts from scratch builds understanding that reading does not. Key projects: implement a maximum likelihood estimator for a Gaussian from data; implement Bayesian updating for a Beta-Binomial model and watch the posterior converge; implement importance sampling for a simple integral; implement a Metropolis-Hastings MCMC sampler for a distribution specified only up to a constant. Each takes a few hours and each builds a piece of the probabilistic machinery that ML frameworks use internally.
PyMC (Python) and Stan are the standard probabilistic programming systems for Bayesian computation, letting you specify probabilistic models declaratively and sample from posteriors automatically. Working through the PyMC tutorials reveals how MCMC connects to Bayesian inference in practice. scipy.stats provides standard distributions and statistical tests for computational practice.
For connecting statistical inference to ML, implement logistic regression by hand using gradient ascent on the log-likelihood, and verify that it matches the cross-entropy loss minimization that PyTorch would compute. This single exercise makes the MLE-cross-entropy-log-likelihood identity concrete and is more convincing than any written explanation.
| Resource | Platform | Type |
|---|---|---|
| Seeing Theory (free) | seeing-theory.brown.edu | Practice |
| PyMC tutorials (free) | docs.pymc.io | Practice |
| Stan documentation (free) | mc-stan.org | Practice |
| scipy.stats documentation (free) | scipy.org | Practice |
| Scratch implementations: MLE, Bayes updating, MCMC | Local | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Confusing probability with likelihood | P(data|θ) is a probability in data when θ is fixed; it is a likelihood in θ when data is fixed. Likelihoods do not integrate to one over θ. Treating likelihoods as probabilities produces wrong interpretations of what MLE optimizes, why cross-entropy works as a loss, and what log-likelihood ratios mean. This confusion is pervasive and consequential. | Derive MLE explicitly for logistic regression: write P(y|x,θ) for binary classification, take the log over the dataset, and maximize. Observe that this is cross-entropy loss up to a sign. This derivation — not an explanation but a derivation — makes the likelihood-probability distinction concrete and permanent. |
| Misreading p-values and confidence intervals | A p-value is the probability of observing data at least as extreme as the actual data, assuming the null hypothesis is true. It is not the probability that the null is false, not the probability that the result will replicate, and not a measure of effect size. A 95% confidence interval is a statement about the procedure that generated it, not a probabilistic statement about the specific interval. Both are routinely misinterpreted in published research. | Study what p-values and confidence intervals actually mean formally — not just the formula but the frequentist interpretation. Then read about the replication crisis to understand what happens when these tools are misunderstood at scale. The Bayesian credible interval, which does say what the confidence interval is mistakenly believed to say, is worth understanding as a contrast. |
| Skipping the continuous case | Many introductory probability courses emphasize discrete probability (coins, dice, counting) and introduce continuous distributions late and briefly. But continuous probability — density functions, integration, the multivariate Gaussian — is where most of ML lives. A learner comfortable with P(X = k) but not with f(x) and ∫f(x)dx will be lost in any derivation involving probability densities, marginals, or expected values computed as integrals. | Do not treat discrete probability as the foundation and continuous probability as an extension. Study both from the start, and ensure comfort with the Gaussian distribution — its density function, its marginal and conditional distributions, sampling from it, and its role in ML — before engaging with probabilistic ML. |
| Treating Bayesian methods as an optional advanced topic | Many ML practitioners learn frequentist statistics first and treat Bayesian methods as something to add later if needed. But many of the most important ML concepts are Bayesian in nature: regularization corresponds to a prior; the MLE-MAP distinction corresponds to the prior-posterior relationship; calibration corresponds to whether posterior predictive probabilities are accurate. Understanding these connections requires fluency in the Bayesian framework. | Study Bayesian inference alongside frequentist inference, not after. McElreath’s approach — starting Bayesian and treating frequentist methods as special cases — is more coherent for ML practitioners than the conventional reverse order. |
| Missing the information-theoretic vocabulary | Entropy, KL divergence, and cross-entropy appear constantly in ML papers, loss functions, and evaluation metrics. Learners who encounter these without the information-theoretic background treat them as arbitrary formulas. Understanding that cross-entropy loss is negative log-likelihood, that KL divergence measures how far the model distribution is from the data distribution, and that entropy bounds the minimum possible loss changes how training objectives are interpreted and chosen. | Read the relevant chapters of MacKay or Cover-Thomas before engaging with variational autoencoders, diffusion models, or any objective involving KL divergence. The investment is modest — a few chapters — and it makes a large vocabulary of ML theory legible that was previously opaque. |
| Underestimating the role of probability in algorithms | Randomized algorithms, hash tables (via the birthday problem and universal hashing), Bloom filters, reservoir sampling, and streaming algorithms all depend on probabilistic reasoning for their analysis and guarantees. A practitioner who treats probability as belonging only to ML and statistics misses half its applications in core CS. | Study Mitzenmacher and Upfal alongside Blitzstein-Hwang. The two books cover complementary territory — one probabilistic intuition, the other probabilistic computation — and together provide the full range of probabilistic thinking that CS requires. |
中文
概率是不确定性的数学。微积分描述量如何连续变化，线性代数描述量如何线性变换，而概率描述量在随机时如何表现——也就是当结果无法被确定预测，只能根据不同结果发生的可能性来刻画时，量会如何表现。一个概率分布并不说什么一定会发生；它说的是，如果同一种情形重复许多次，每个结果会以怎样的频率出现，或者等价地说，在给定我们所知信息的情况下，我们应该对每种可能性有多大信心。
统计是在不确定性下从数据中得出结论的学科。概率模型描述一个数据生成过程；统计则反向进行推断：给定数据，我们能对生成这些数据的过程得出什么结论？这包括估计未知参数，检验观察到的模式更可能是真实存在而非偶然产生的，以及量化结论中的不确定性。这两个主题不可分割：统计需要概率来定义“可能”和“有信心”是什么意思，而概率本身在数据进入之前并不会告诉你什么。
信息论是把概率应用于对不确定性本身的度量。它问的是：一个概率分布代表多少不确定性？观察一个变量能告诉你多少关于另一个变量的信息？平均来说，编码来自某个给定分布的消息需要多少 bit？其中的核心量——熵、互信息、KL 散度——作为损失函数、正则化项和评估指标反复出现在机器学习中，尽管它们常常并不被标注为信息论。
这三个主题在多数课程中被分开教授，但在实践中是统一的。机器学习同时建立在三者之上：语言模型定义一个概率分布，通过最小化统计损失（交叉熵）来训练，并通过信息论指标（困惑度）来评估。理解每个概念从哪里来、它实际在说什么，可以避免把它们孤立学习时产生的混乱。
前置知识：微积分（§2.4）是连续概率所必需的，尤其是概率密度函数，以及作为积分的期望值。离散概率（§2.2）是有限情形下的特殊情况。本节是第五章（人工智能）和 §4.5 部分内容的前置基础。
从赌桌到学习的数学
概率的形式化研究始于赌博。1654 年夏天，Blaise Pascal 和 Pierre de Fermat 就一个赌徒提出的问题通信：如果一场机会游戏在结束前被中断，赌注应该如何分配？这个问题要求人们对尚未发生的事件进行推理——也就是给各种可能性赋予数字。Pascal 与 Fermat 的通信确立了基础规则：如何计数等可能结果，如何计算复合事件的概率，如何把期望值作为公平下注的指南。他们没有形式框架；他们只是在解决具体问题。但这些问题迫使他们发展出工具，而这些工具后来被证明具有普遍性。
在十七、十八世纪，概率从赌博向外扩展到自然哲学、人口统计学和误差理论。Jacob Bernoulli 在 1713 年证明了第一个极限定理：如果你把一个实验重复许多次，观察到的频率会收敛到真实概率。这个大数定律说明了一件很深的事——个体结果中的随机性，可以与总体行为中的规律性并存；概率具有经验意义，而不只是记账约定。Abraham de Moivre 在 1733 年发现，许多独立随机事件之和会趋向钟形曲线——这是中心极限定理的先驱，而 Pierre-Simon Laplace 会在 1812 年给出其完全一般的证明。Carl Friedrich Gauss 在 1809 年使用钟形曲线，为最小二乘法作为组合带随机误差的天文观测的最优方法提供正当性，从而把概率与科学实践连接起来。
十九世纪加入了统计学。Adolphe Quetelet 在 1830 和 1840 年代把概率应用到社会数据中，表明犯罪率、出生率和群体身体测量都遵循稳定的统计模式——由此开创了量化社会科学。Francis Galton 在 1880 年代发明了回归概念，发现高个父母的子女身高往往会向群体平均值回归，而不是保持父母的极端值。Karl Pearson 系统化了 Galton 的思想，并发明了卡方检验和相关系数，为统计学提供了第一批形式化假设检验工具。到 1900 年，概率和统计已经成为一门同时拥有数学基础和经验应用的统一学科——但其基础仍然是非形式的。概率究竟是什么？是重复试验中的频率？还是信念程度？答案此前一直被隐含处理。
二十世纪提供了两个不相容的答案。Andrey Kolmogorov 1933 年的专著 Grundbegriffe der Wahrscheinlichkeitsrechnung——“概率论基础”——把概率放到了与数学其他分支相同的公理化基础上。公理很简单：概率非负，整个样本空间的概率为一，互斥事件并集的概率等于它们概率之和。从这三条公理出发，整个概率论都可以被严谨推出。Kolmogorov 的公理并不说明概率意味着什么，只说明它如何表现。而“概率意味着什么”这个问题，引发了一场至今尚未解决的哲学争论。
Ronald Fisher 是二十世纪最有影响力的统计学家，他支持频率主义解释：概率是某个事件在无限独立重复序列中的极限频率。这使概率变得客观、可经验检验，但也让某些概率陈述变得别扭——例如，一个特定未知参数落在某个特定区间中的概率，在频率主义视角下要么是零，要么是一，因为参数虽然未知，但它是固定的。Fisher 在这个框架中发展了最大似然估计、p 值和方差分析，创造了主导二十世纪大部分经验科学的方法论。
Bruno de Finetti 和 Frank Ramsey 在 1920 年代末和 1930 年代独立工作，捍卫贝叶斯解释：概率是信念程度，会根据证据通过 Bayes 定理进行一致更新。在这个视角下，给一个特定参数赋予概率是完全有意义的——它表达的是在给定已有数据时，我们对该参数取值的信心。Harold Jeffreys 在 1930 和 1940 年代发展了客观贝叶斯方法，试图定义表达无知而非任意信念的先验分布。贝叶斯—频率主义分歧，成为二十世纪统计学的定义性智力断层。
这条断层部分被计算消解了，而不是被哲学论证消解。MCMC——马尔可夫链蒙特卡洛——使贝叶斯推断在计算上变得可处理。Metropolis 算法由 Los Alamos 的一个团队在 1953 年发展出来，团队成员包括 Nicholas Metropolis 和 Edward Teller，最初用于物理仿真；Gelfand 和 Smith 1990 年的论文展示了如何把它应用到任意贝叶斯后验分布上。此前贝叶斯推断主要局限于具有共轭先验的问题（后验与先验具有相同形式），而 MCMC 使人们可以从任意复杂的后验中采样。与不断增长的计算能力结合后，它在 1990 和 2000 年代为贝叶斯方法进入统计学、机器学习和科学计算打开了大门。
信息论则来自完全不同的方向。Claude Shannon 1948 年的论文 “A Mathematical Theory of Communication” 问的是：一条消息包含多少信息？Shannon 把一个概率分布的熵定义为 H = -Σ p(x) log p(x)——这是对分布平均不确定性的度量。高熵意味着许多结果几乎同样可能；低熵意味着大部分概率集中在少数结果上。Shannon 的基本定理证明，熵是编码来自某个给定分布的消息所需的最小平均 bit 数，从而把对不确定性的抽象度量与高效通信的具体问题连接起来。KL 散度——当一个分布更新为另一个分布时获得的信息——也在同一篇论文中作为衡量两个分布差异的量出现。交叉熵则表现为：当你用为分布 q 优化的编码来编码来自分布 p 的消息时，平均编码长度是多少。最小化交叉熵等价于最大化数据在模型下的似然，因此交叉熵损失和最大似然估计，是同一个目标的两个名字。
Shannon 的信息论和 Kolmogorov 的概率论是独立发展出来的，但二者结合后成为机器学习的理论基础。语言模型训练是最大似然估计，也就是最小化交叉熵，也就是最小化模型分布与数据分布之间的 KL 散度。分类器的泛化能力受 Rademacher 复杂度约束，而这是一个概率量。神经网络学到的表示可以通过信息瓶颈来分析，而信息瓶颈是一个互信息框架。统计学习理论研究机器学习算法何时、为何能够从训练数据泛化到新数据，它全程使用概率论，其核心结果（VC 维、PAC 界）使用组合概率证明。当代机器学习在数学核心上，是由计算增强的概率与统计分支。
条件化、估计与不确定性的度量
条件化：基于证据更新的逻辑
| 条件概率是概率推理的核心概念。P(A | B)——“在 B 给定时 A 的概率”——捕捉的是当已知另一个事件发生时，一个事件的概率如何变化。它的定义很简单：P(A | B) = P(A ∩ B)/P(B)。但这个概念远不止一个公式。它是对“知道 B 会告诉我关于 A 的什么？”这一问题的正确回答，并支配所有贝叶斯推理。 |
| Bayes 定理可以由这个定义通过简单代数推出：P(θ | data) = P(data | θ)P(θ)/P(data)。似然 P(data | θ) 度量的是：如果参数是 θ，观察到的数据有多大概率出现。先验 P(θ) 捕捉的是在看到数据之前关于 θ 的初始信念。后验 P(θ | data) 是看到数据之后更新后的信念。分母 P(data) 是归一化常数，确保后验积分为一。这就是贝叶斯学习的完整逻辑：信念由证据更新，而似然函数在先验和后验之间起中介作用。 |
| 最大似然估计（MLE）是带均匀先验的贝叶斯估计的特殊情形：在所有可能参数值中，选择让观察数据最可能出现的那个。这等价于选择最大化 P(data | θ) 的参数，而这个量就是似然。值得注意的是，用最小化交叉熵损失来训练神经网络，在分类分布下正是 MLE：模型学习的是让训练数据在其概率模型下最可能出现的参数。这个连接——交叉熵损失等于负对数似然，等于 MLE 目标——不是类比，而是恒等式。 |
| 独立性假设会极大简化条件化。两个事件 A 和 B 独立，意味着 P(A | B) = P(A)：知道 B 不会告诉你任何关于 A 的信息。独立性允许联合概率分解：P(A ∩ B) = P(A)P(B)。在实践中，真正的独立性很少见，但条件独立性——在给定 C 时，A 与 B 独立——是概率图模型、朴素 Bayes 分类器和 Markov 链背后的结构性假设。概率链式法则把任意联合分布表示为条件概率的乘积：P(X₁, …, Xₙ) = P(X₁)P(X₂ | X₁)…P(Xₙ | X₁,…,Xₙ₋₁)。这也是语言模型使用的自回归分解：P(word₁, word₂, …) = P(word₁)·P(word₂ | word₁)·P(word₃ | word₁,word₂)···。生成 GPT 的架构，从概率角度看，就是建立在这种分解之上的自回归模型。 |
中心极限定理是连接条件化与总体行为的伟大极限定理。当你从一个具有有限均值和方差的分布中取许多独立样本并求平均时，这个平均值的分布会趋近 Gaussian，而不管原始分布是什么。收敛速度是 O(1/√n)：对 n 个样本求平均，会把估计的标准差降低 √n 倍。这个定理解释了为什么频率主义置信区间具有现在这种形式，为什么在 Gaussian 噪声下最小二乘是最优的，以及为什么大数定律以它所表现的方式成立——它是从样本进行统计推断的数学基础。
统计推断：读懂数据
统计推断是在有限数据中得出可辩护结论的学科。它处在中心极限定理的阴影下运作：不管底层分布是什么，大样本都会以可预测方式表现，而这种可预测性正是统计力量的来源。
估计问的是：给定数据，对于一个未知量，最好的猜测是什么？点估计给出一个数字；区间估计给出一个可能值范围。频率主义置信区间经常被误解，它其实是关于程序的陈述，而不是关于某个特定区间的陈述：如果你用这种方法在许多独立数据集上构造置信区间，其中 95% 会包含真实参数。它不是说真实参数以 95% 的概率落在这个特定区间中——在频率主义视角下，真实参数是固定的，而这个具体区间要么包含它，要么不包含它。相比之下，贝叶斯可信区间是关于概率的陈述：给定数据和先验，参数有 95% 的概率落在这个范围内。两种区间在数值上常常相近，但逻辑上不同。
假设检验问的是：观察到的效应是真实的，还是在“无效应”这一零假设下也合理可能出现？Fisher 的 p 值是在零假设为真时，观察到至少与实际数据一样极端的数据的概率。小 p 值是反对零假设的证据；它不是支持备择假设的证据，也不是零假设为假的概率。Neyman-Pearson 框架把这形式化为决策规则：设定显著性水平 α，如果 p 值低于 α，则拒绝零假设，并接受这一程序会以 α 的比例产生错误拒绝（I 类错误）。对 p 值实际含义的广泛误解，在很大程度上促成了 2010 年后出现的可重复性危机；当时大规模重复实验表明，心理学、医学和社会科学中许多已发表发现无法复现。
混杂与因果性是统计学中最深的困难。两个变量之间的相关——比如吸烟与肺癌——可能反映因果效应，也可能由一个同时导致二者的混杂变量解释。无论样本多大，观察数据本身都无法区分相关和因果；只有实验设计（随机分配到处理组和对照组）或编码在图中的因果假设，才能让因果结论变得可辩护。Pearl 的 do-calculus 以及 Rubin 和 Holland 的潜在结果框架，为因果推断提供了形式工具。这对 ML 很重要，因为在观察数据上训练出的模型学到的是相关关系；当数据生成过程改变时，相关可能断裂，而因果结构仍然稳定。
信息论：度量不确定性
熵量化一个概率分布的不确定性。对离散分布来说，H(X) = -Σ p(x) log₂ p(x) bit。确定性分布——某个结果概率为一——熵为零：没有不确定性。n 个结果上的均匀分布拥有 log₂ n bit 的熵：不确定性最大。熵度量观察到一个结果时的平均惊讶程度：罕见结果比常见结果更令人惊讶，而 Shannon 把惊讶定义为 -log p(x)。
| KL 散度 D_KL(P | Q) = Σ p(x) log(p(x)/q(x)) 度量分布 Q 与分布 P 有多不同。它不是对称的——通常 D_KL(P | Q) ≠ D_KL(Q | P)——而这种不对称是有意义的。前向 KL（P | Q）在 Q 在 P 赋予高概率的地方也赋予高概率时较小；最小化前向 KL 会产生覆盖模式的行为。反向 KL（Q | P）在 Q 集中于 P 较大的区域时较小；最小化反向 KL 会产生寻模行为。变分自编码器最小化的是前向 KL 的一个界。许多其他生成模型训练目标，都可以被理解为在前向 KL 与反向 KL 之间作出的选择。 |
| 交叉熵 H(P, Q) = -Σ p(x) log q(x) 度量的是：当来自 P 的消息用为 Q 优化的编码来编码时，平均编码长度是多少。它可以分解为 H(P, Q) = H(P) + D_KL(P | Q)：交叉熵等于真实分布的熵，加上真实分布与模型分布之间的 KL 散度。当 P 是数据分布、Q 是模型时，最小化交叉熵等价于最小化从数据分布到模型的 KL 散度——也就是让模型尽可能接近真实数据分布。这正是每一个语言模型、图像生成模型和密度估计器的目标。 |
| 互信息 I(X; Y) = H(X) - H(X | Y) 度量知道 Y 会减少多少关于 X 的不确定性。它是对称的——I(X;Y) = I(Y;X)——并且当且仅当 X 和 Y 独立时等于零。互信息出现在特征选择中（选择与目标具有高互信息的特征）、表征学习中（信息瓶颈原则），也出现在对神经网络注意力模式的分析中。与相关系数不同，互信息能够捕捉非线性依赖。 |
学习这一部分会改变什么
概率和统计会改变实践者解释证据与不确定性的方式，而不只是改变他们如何计算。
最重要的变化是认识论上的：条件推理的纪律。受过概率训练的实践者在评估一个主张时，会自动追问先验概率是多少、似然比是多少、什么证据会改变自己的信念。这种贝叶斯认识论并不局限于形式计算——它是一种思维方式，用来区分谨慎的经验推理和动机性推理。科学中的可重复性危机，很大程度上正是这种思维方式的失败：在小样本和弱零假设下看起来统计显著的效应，被发表为已确立发现；而当某个特定效应的先验概率很低、样本量又小时，一个显著 p 值所提供的证据远比它看起来要弱。
第二个变化，是设计和评估实验的能力。因果主张需要因果结构——要么来自随机化，要么来自显式因果假设——而不仅仅是统计关联。一个吸收了观察数据与实验数据差异、理解混杂和选择偏差、知道随机化能带来什么的实践者，才能评估某个声称的结果是否真的得到了证据支持。这是任何收集数据的领域进行有意义经验工作的前提，而 CS 的多数领域都在收集数据。
第三个变化，是掌握机器学习的概率词汇。损失函数是对数似然。正则化是先验。变分自编码器的训练目标是对数似然的下界。Score matching 等价于最小化 Fisher 散度。对比学习优化的是似然比。这些连接不是隐喻；它们是恒等式。理解它们，会改变哪些架构和算法选择是可解释的，哪些只是临时拼凑的。
第四个变化，是校准过的不确定性。没有统计训练的实践者，倾向于把模型输出当作事实。受过统计训练的实践者则会问：这个估计的置信区间是什么？这个模型是否校准——它的置信度分数是否匹配经验频率？要可靠检测这个效应，需要多大的样本量？这些问题都有明确答案，而提出并回答这些问题的能力，正是科学化 ML 实践与工程直觉之间的区别。
资源
书籍与文本
Blitzstein 和 Hwang 的 Introduction to Probability（第 2 版，2019）是概率论最好的单本入口。它的独特之处在于教学上的细致：每个概念都通过例子发展，直觉与形式并行建立，而条件推理会贯穿始终，而不是被当作特殊主题处理。其中的 “story proofs”——先于形式证明的直觉论证——训练的不只是技巧，也是概率直觉。Blitzstein 的 Harvard Stat 110 讲座（YouTube 免费）同样优秀，有时甚至更好；他的现场教学会以文字偶尔压缩掉的方式展现数学背后的推理。多数学习者应把书和讲座结合使用。
Wasserman 的 All of Statistics: A Concise Course in Statistical Inference（2004）是数学准备较好的 CS 学习者进入统计推断的最高效路径。它在约 450 页中覆盖点估计、置信区间、假设检验、回归、非参数方法和贝叶斯推断——远短于传统统计教材——同时并未牺牲严谨性。简洁正是重点：Wasserman 覆盖了 CS 实践者所需内容，而没有用大量应用例子把传统统计教材扩展到 800 页。先读 Blitzstein-Hwang，再读 Wasserman，是 CS 实践者从概率基础进入统计推断的最直接路径。
对于兼具深度和可及性的贝叶斯推断，McElreath 的 Statistical Rethinking（第 2 版，2020，配套免费讲座）是目前最有效的处理。McElreath 把贝叶斯推理作为一种从数据中学习的连贯方式来教授，而不是把它当作在学完频率主义方法后再补充的技术。他坚持画生成模型，从假设推导似然，并计算后验，而不是查检验统计量——这些习惯会直接迁移到概率机器学习中。这本书在一个统一的贝叶斯框架中覆盖回归、多层模型和因果推断。对于想在推断层面理解概率 ML 在做什么的实践者来说，这是正确伴读材料。
对于信息论，Cover 和 Thomas 的 Elements of Information Theory（第 2 版，2006）是标准严谨处理。它从第一原则出发覆盖熵、互信息、信道容量、源编码和率失真理论。MacKay 的 Information Theory, Inference, and Learning Algorithms（可在 inference.org.uk 免费获取）覆盖类似内容，但以 Cover-Thomas 所没有的方式，把它与贝叶斯推断和机器学习统一起来。MacKay 这本书会揭示为什么交叉熵损失和最大似然估计是同一件事，为什么 KL 散度出现在变分推断中，以及为什么熵正则化是合理的。对 ML 实践者来说，MacKay 是更直接有用的入口；对信息论研究者来说，Cover-Thomas 是更全面的参考。
对于概率机器学习本身，Murphy 的 Probabilistic Machine Learning: An Introduction（2022，可在 probml.ai 免费获取）是当代综合参考，在统一框架中同时覆盖概率基础和 ML 模型。Bishop 的 Pattern Recognition and Machine Learning（2006）仍然是定义了概率 ML 方法的经典参考。二者都适合作为深入处理的参考，而不是第一本书。
对于具有算法应用的CS 专门概率——随机算法、哈希函数、概率数据结构、集中不等式——Mitzenmacher 和 Upfal 的 Probability and Computing（第 2 版，2017）是正确教材。它从 CS 视角覆盖概率方法、Markov 链、随机图和流式算法，而这些内容不是 Blitzstein-Hwang 或 Wasserman 的重点。想同时获得概率基础和算法应用的学习者，应先读 Blitzstein-Hwang，再读 Mitzenmacher-Upfal。
对于统计学习理论——对学习算法何时以及为何能够泛化的数学分析——Shalev-Shwartz 和 Ben-David 的 Understanding Machine Learning（免费在线）是可进入的入口，以异常清晰的方式覆盖 PAC 学习、VC 维和偏差—复杂度权衡。Vershynin 的 High-Dimensional Probability（免费在线）覆盖集中不等式，而这在理解现代过参数化模型时越来越关键。
对于严谨概率基础，Williams 的 Probability with Martingales 是测度论概率的标准可进入导论。它在约 250 页中发展 Lebesgue 积分和鞅理论，并且极其清晰——对于想理解数学基础，而不只是掌握计算工具的学习者，这是合适入口。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Blitzstein & Hwang, Introduction to Probability（第 2 版） | 经典概率入口；条件推理 | 入门 |
| Wasserman, All of Statistics（2004） | 面向 CS 的简明统计推断 | 深入 |
| McElreath, Statistical Rethinking（第 2 版） | 贝叶斯推理习惯；生成模型 | 深入 |
| Cover & Thomas, Elements of Information Theory（第 2 版） | 信息论参考 | 参考 |
| MacKay, Information Theory, Inference, and Learning Algorithms（免费） | 统一概率、推断与 ML | 深入 |
| Mitzenmacher & Upfal, Probability and Computing（第 2 版） | 面向算法和 CS 的概率 | 深入 |
| Shalev-Shwartz & Ben-David, Understanding Machine Learning（免费） | 统计学习理论 | 深入 |
| Vershynin, High-Dimensional Probability（免费） | 集中不等式；现代理论 | 深入 |
| Gelman et al., Bayesian Data Analysis（第 3 版） | 综合贝叶斯参考 | 参考 |
| Williams, Probability with Martingales | 测度论基础 | 深入 |
| Murphy, Probabilistic Machine Learning: An Introduction（免费） | 综合概率 ML | 辅助 |
| Bishop, Pattern Recognition and Machine Learning | 经典概率 ML 参考 | 辅助 |
| Koller & Friedman, Probabilistic Graphical Models | 图模型参考 | 辅助 |
课程与讲座
Blitzstein 的 Harvard Stat 110（YouTube 免费）是目前可免费获取的最精致概率课程。Blitzstein 的教学风格通过完整例题和对常见错误的细致关注来建立概率直觉。把讲座与教材一起观看，效果显著强于单独使用其中任何一个；他的现场推理往往比书面推导更有揭示性。
MIT 6.041/6.431（Probability，MIT OCW 免费）是 MIT 的概率课程，严谨而全面。它与 Stat 110 的侧重点不同——更形式化，不那么以直觉优先——可以作为很好的补充。完整讲课视频、习题和考试都可获取。
McElreath 的 Statistical Rethinking 讲座（YouTube 免费）逐章配合教材。讲座通过 R 和 Stan 中的完整例子发展贝叶斯推理习惯，使抽象概念更具体。2023 年版本的讲座更新了例子，并包含关于模型检查和因果推断的当代视角。
MIT 18.650（Statistics for Applications，MIT OCW 免费）以研究生水平覆盖统计推断——估计、假设检验、回归、贝叶斯方法。它与 Stat 110 的概率重点互补。
Berkeley Data 8 及其免费教材 Computational and Inferential Thinking 构成了从编程到统计的强应用桥梁。这门课使用 Jupyter notebook、真实数据集、仿真和推断，非常适合那些需要把统计变成可以计算之物，而不仅是可以推导之物的学习者。它不应替代概率/统计主干，但它是最好的应用伴读之一。
StatQuest with Josh Starmer（YouTube 免费，附可选付费材料）为概率、统计和机器学习概念提供短小的视觉解释：p 值、回归、逻辑回归、Bayes、随机森林、PCA 及相关主题。它的主要价值是修复理解和建立直觉，而不是深度；当一个概念在课程或教材解释之后仍然不透明时，可以使用它。
| 课程 | 平台 | 类型 |
|---|---|---|
| Blitzstein, Stat 110 Probability（免费） | YouTube / Harvard 课程网站 | 入门 |
| MIT 6.041 Probability（免费） | MIT OCW | 入门 |
| McElreath, Statistical Rethinking lectures（免费） | YouTube | 深入 |
| MIT 18.650 Statistics for Applications（免费） | MIT OCW | 深入 |
| Berkeley Data 8 + Computational and Inferential Thinking（免费） | Berkeley / Jupyter Book | 实践 |
| StatQuest with Josh Starmer（免费；付费材料可选） | YouTube / statquest.org | 辅助 |
实践、工具与代码
Seeing Theory（seeing-theory.brown.edu，免费）是一个交互式概率和统计视觉入门。它交互式展示概率分布、期望值、大数定律、置信区间和中心极限定理——对于那些很难只靠文本建立直觉的概念，它是合适补充。
从零实现概率概念，会建立阅读无法带来的理解。关键项目包括：从数据中实现 Gaussian 的最大似然估计；实现 Beta-Binomial 模型的贝叶斯更新，并观察后验如何收敛；为一个简单积分实现重要性采样；为一个只给定到归一化常数的分布实现 Metropolis-Hastings MCMC 采样器。每个项目需要几小时，每个项目都会建立机器学习框架内部使用的一部分概率机制。
PyMC（Python）和 Stan 是贝叶斯计算的标准概率编程系统，允许你以声明式方式指定概率模型，并自动从后验中采样。学习 PyMC 教程，会揭示 MCMC 如何在实践中连接到贝叶斯推断。scipy.stats 提供标准分布和统计检验，是计算实践中的常用工具。
要把统计推断连接到 ML，可以手写实现逻辑回归：对对数似然做梯度上升，并验证它与 PyTorch 所计算的交叉熵损失最小化相匹配。这个单一练习会让 MLE—交叉熵—对数似然之间的恒等式变得具体，比任何文字解释都更有说服力。
| 资源 | 平台 | 类型 |
|---|---|---|
| Seeing Theory（免费） | seeing-theory.brown.edu | 实践 |
| PyMC tutorials（免费） | docs.pymc.io | 实践 |
| Stan documentation（免费） | mc-stan.org | 实践 |
| scipy.stats documentation（免费） | scipy.org | 实践 |
| 从零实现：MLE、Bayes 更新、MCMC | 本地 | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 | ||
|---|---|---|---|---|
| 混淆概率和似然 | 当 θ 固定时，P(data | θ) 是关于 data 的概率；当 data 固定时，它是关于 θ 的似然。似然不会在 θ 上积分为一。把似然当成概率，会导致对 MLE 优化什么、交叉熵为什么能作为损失，以及对数似然比是什么意思的错误理解。这种混淆很普遍，而且后果严重。 | 明确为逻辑回归推导 MLE：为二分类写出 P(y | x,θ)，对整个数据集取对数，然后最大化。观察到这与交叉熵损失只差一个符号。这个推导——不是解释，而是推导——会让似然与概率的区别具体且持久。 |
| 误读 p 值和置信区间 | p 值是在零假设为真时，观察到至少与实际数据同样极端的数据的概率。它不是零假设为假的概率，不是结果会复现的概率，也不是效应大小的度量。95% 置信区间是关于生成它的程序的陈述，而不是关于这个特定区间的概率陈述。二者在已发表研究中都经常被误解。 | 学习 p 值和置信区间形式上真正意味着什么——不只是公式，而是频率主义解释。然后阅读可重复性危机，理解这些工具在大规模被误解时会发生什么。贝叶斯可信区间值得作为对照来理解，因为它说的正是人们常常误以为置信区间在说的东西。 | ||
| 跳过连续情形 | 许多入门概率课程强调离散概率（硬币、骰子、计数），并在很晚且很简略地引入连续分布。但连续概率——密度函数、积分、多元 Gaussian——才是大多数 ML 所在之处。一个只熟悉 P(X = k)，却不熟悉 f(x) 和 ∫f(x)dx 的学习者，在任何涉及概率密度、边缘分布或作为积分计算的期望值推导中都会迷失。 | 不要把离散概率当作基础，把连续概率当作扩展。从一开始就同时学习二者，并确保在进入概率 ML 之前熟悉 Gaussian 分布——它的密度函数、边缘分布和条件分布、从中采样的方法，以及它在 ML 中的作用。 | ||
| 把贝叶斯方法当成可选高级主题 | 许多 ML 实践者先学习频率主义统计，再把贝叶斯方法当作之后有需要时再补充的内容。但许多最重要的 ML 概念本质上是贝叶斯的：正则化对应先验；MLE 与 MAP 的区别对应先验—后验关系；校准对应后验预测概率是否准确。理解这些连接需要熟悉贝叶斯框架。 | 把贝叶斯推断与频率主义推断并行学习，而不是之后再学。McElreath 的方式——从贝叶斯开始，把频率主义方法视为特殊情形——对 ML 实践者来说，比传统的反向顺序更连贯。 | ||
| 缺失信息论词汇 | 熵、KL 散度和交叉熵在 ML 论文、损失函数和评估指标中不断出现。没有信息论背景的学习者会把它们当成任意公式。理解交叉熵损失就是负对数似然，KL 散度度量模型分布与数据分布之间的距离，熵限定了可能的最小损失，会改变人们解释和选择训练目标的方式。 | 在接触变分自编码器、扩散模型，或任何涉及 KL 散度的目标之前，阅读 MacKay 或 Cover-Thomas 的相关章节。投入并不大——几章而已——但它会让大量原本不透明的 ML 理论词汇变得可读。 | ||
| 低估概率在算法中的作用 | 随机算法、哈希表（通过生日问题和 universal hashing）、Bloom filters、reservoir sampling 和流式算法，都依赖概率推理来进行分析和给出保证。把概率只看成属于 ML 和统计的实践者，会错过它在核心 CS 中一半的应用。 | 把 Mitzenmacher 和 Upfal 与 Blitzstein-Hwang 一起学习。这两本书覆盖互补地带——一本训练概率直觉，另一本处理概率计算——合在一起才能提供 CS 所要求的完整概率思维范围。 |