English
Information is what resolves uncertainty. When you receive a message, the information it contains is precisely the amount by which it reduces your uncertainty about the world. This definition sounds circular until you ask: can uncertainty be measured? Claude Shannon’s answer, worked out in a 1948 paper that founded an entire discipline, was yes — and the measure has a specific form that follows necessarily from a small number of constraints about what a measure of uncertainty should look like.
The entropy of a probability distribution P over outcomes is H(P) = -Σ p(x) log₂ p(x), measured in bits. A fair coin has entropy 1 bit: it takes one yes/no question to determine the outcome. A fair die has entropy log₂ 6 ≈ 2.58 bits. A deterministic outcome has entropy 0 bits: there is no uncertainty to resolve. This is not a convention but a theorem: entropy is the unique function satisfying natural axioms about what a measure of uncertainty should be, up to a positive multiplicative constant. The definition captures something real.
For computer science, information theory provides three things. First, a quantitative language for describing uncertainty, redundancy, and compression — every lossless compression algorithm is bounded by the entropy rate of its source, and efficient algorithms approach this bound. Second, the mathematical foundation for reasoning about communication over noisy channels — Shannon’s channel capacity theorem tells you the maximum rate at which information can be reliably transmitted, regardless of the coding scheme. Third, a unifying vocabulary for machine learning — cross-entropy loss is information-theoretic, KL divergence is information-theoretic, the evidence lower bound in variational autoencoders is information-theoretic, and mutual information underlies many modern representation learning objectives. Understanding information theory makes the mathematics of machine learning interpretable rather than opaque.
Prerequisites: Probability and statistics (§2.5) — entropy, KL divergence, and mutual information are all defined in terms of probability distributions, and the source and channel coding theorems are probabilistic results. Calculus (§2.4) is needed for the continuous case.
How the Concept of Information Became Precise
Before 1948, “information” was an everyday word with no mathematical meaning. Engineers working on telephone and telegraph systems knew that signals carried information and that noise degraded it, but the concept of information itself was vague — it could not be measured, compared, or optimized. Claude Shannon’s 1948 paper “A Mathematical Theory of Communication,” published in the Bell System Technical Journal, changed this in a single paper that is among the most consequential in the history of science.
Shannon’s starting question was practical: how much can you compress a message without losing content, and how fast can you reliably transmit information over a noisy channel? These had seemed like engineering questions requiring engineering answers. Shannon showed they were mathematical questions with mathematical answers — answers that held regardless of the particular encoding, the particular signal, or the particular noise, as long as the statistics were specified.
The mathematical predecessors were scattered and partial. Harry Nyquist had derived the sampling theorem in 1928, establishing that a band-limited signal could be perfectly reconstructed from discrete samples at twice the bandwidth — a result that is now understood as a special case of Shannon’s framework. Ralph Hartley had proposed a logarithmic measure of information capacity in 1928. Norbert Wiener, working independently on cybernetics and signal processing, was developing similar ideas at MIT during the same years Shannon was working at Bell Labs. Shannon knew and built on this prior work, but his synthesis — particularly the channel capacity theorem and the noisy channel coding theorem — went far beyond anything his predecessors had achieved.
The entropy definition emerged from a uniqueness argument. Shannon asked: what properties should a measure of uncertainty have? It should be continuous in the probabilities. It should be maximized when all outcomes are equally likely. And it should satisfy a consistency condition: the uncertainty of a joint outcome should equal the uncertainty of the first outcome plus the expected additional uncertainty given the first. These axioms uniquely determine entropy up to a constant (Shannon, 1948; Khinchin gave a more rigorous proof in 1957). The logarithm base determines the units: base 2 gives bits, base e gives nats, base 10 gives dits. The choice is convention; the function is not.
The source coding theorem established the fundamental limit of lossless compression. Given a source that produces symbols drawn independently from a distribution P, no lossless compression scheme can compress the source to fewer than H(P) bits per symbol on average — and there exist schemes that approach this limit as closely as desired. The coding scheme Shannon proved was constructive but not efficient; practical schemes came later. David Huffman, then a graduate student at MIT, derived an optimal prefix-free code in 1952 as a course assignment after being told he could skip the final exam if he found a better construction than the existing methods. The Huffman algorithm is optimal for symbol-by-symbol coding; arithmetic coding, developed through the 1970s and 1980s, approaches the entropy rate for longer sequences and underpins modern lossless compressors including the DEFLATE format used in gzip and PNG.
The noisy channel coding theorem was the deeper and more surprising result. A channel with noise — one where transmitted symbols may be corrupted before they arrive — has a capacity C, measured in bits per channel use. If you transmit at rate R < C, there exist coding schemes that achieve arbitrarily small error probability. If you transmit at rate R > C, errors are unavoidable no matter what coding scheme you use. The theorem is an existence proof: Shannon showed such codes must exist (by a probabilistic argument — randomly chosen codes achieve the bound with high probability) without constructing them. Finding practical codes that approach channel capacity took decades more. The Hamming codes (1950) were the first algebraic error-correcting codes. Reed-Solomon codes (1960) are used in CDs, DVDs, and space communications. Turbo codes (1993) and LDPC codes (rediscovered in the 1990s from Gallager’s 1962 thesis) came within fractions of a decibel of the Shannon limit. Polar codes (Arıkan, 2009) are the first codes proven to provably achieve channel capacity for any symmetric binary-input channel and are used in 5G communications.
| The information-theoretic framework spread beyond its original communications context through a convergence with statistics and probability. Solomon Kullback and Richard Leibler formalized what is now called KL divergence in 1951: D_KL(P | Q) = Σ p(x) log(p(x)/q(x)) measures the information gain when updating from distribution Q to distribution P. KL divergence appears everywhere in statistical inference: the maximum likelihood estimator minimizes KL divergence from the data distribution to the model; Bayesian updating can be understood as minimizing KL divergence from the prior to the posterior; the principle of maximum entropy (Jaynes, 1957) selects the distribution that maximizes entropy subject to constraints, justifying the exponential family as the class of distributions with sufficient statistics. |
Andrey Kolmogorov, Gregory Chaitin, and Ray Solomonoff independently developed algorithmic information theory in the 1960s — a complexity measure for individual strings rather than distributions. The Kolmogorov complexity K(x) of a string x is the length of the shortest program that outputs x. Unlike Shannon entropy (which is a property of a distribution), Kolmogorov complexity is a property of individual strings, but the two are deeply connected: for a random variable X drawn from distribution P, the expected Kolmogorov complexity is approximately the Shannon entropy H(P). Algorithmic information theory provides foundations for minimum description length (MDL) — a model selection principle that trades off model complexity against goodness of fit — and illuminates the relationship between compression and statistical inference.
| The contemporary period has seen information theory become a foundational language for machine learning. Cross-entropy loss, used to train nearly every classification model, is the expected log-probability of the correct label: H(P, Q) = -Σ p(x) log q(x), the cross-entropy between the data distribution P and the model Q. This is not just an arbitrary loss function — it is the KL divergence from P to Q plus the entropy of P, and minimizing it is equivalent to minimizing the KL divergence, which is equivalent to maximum likelihood estimation. The evidence lower bound (ELBO) in variational autoencoders decomposes as a reconstruction term minus a KL divergence, making the VAE training objective explicitly information-theoretic. Mutual information I(X;Y) = H(X) - H(X | Y) = H(Y) - H(Y | X) has become the objective of choice for self-supervised representation learning methods (mutual information maximization, InfoNCE, contrastive learning variants) that learn representations without labels. The information bottleneck method (Tishby, Pereira, Bialek, 1999 and reinvestigated by Tishby and Schwartz-Ziv in 2017) proposes that deep learning works by compressing input information while preserving task-relevant information — a framework that generated significant debate about whether it accurately describes what neural networks do. Whether or not that specific claim is correct, the information-theoretic vocabulary for discussing what representations contain is now standard. |
Entropy, Channels, and the Geometry of Distributions
Entropy and Its Extensions
Entropy H(X) = -Σ p(x) log p(x) is the fundamental quantity, but several extensions are needed for reasoning about multiple variables and about divergence between distributions.
| Joint entropy H(X, Y) = -Σ_{x,y} p(x,y) log p(x,y) measures the total uncertainty in a pair of variables. Conditional entropy H(X | Y) = H(X,Y) - H(Y) measures the remaining uncertainty in X after Y is observed. The chain rule H(X,Y) = H(X) + H(Y | X) decomposes joint entropy into marginal plus conditional, generalizing straightforwardly to n variables. |
| Mutual information I(X;Y) = H(X) - H(X | Y) = H(Y) - H(Y | X) = H(X) + H(Y) - H(X,Y) measures the reduction in uncertainty about X from observing Y, and symmetrically the reduction about Y from observing X. It is always non-negative and equals zero if and only if X and Y are independent. It is not a metric (the triangle inequality fails) but it is symmetric and it captures nonlinear dependencies, unlike correlation. For continuous variables, differential entropy h(X) = -∫ p(x) log p(x) dx extends the definition but loses some properties of discrete entropy — differential entropy can be negative, and it is not invariant under reparameterization. |
The data processing inequality is one of the most practically useful consequences of information theory: if X → Y → Z is a Markov chain (Z is conditionally independent of X given Y), then I(X;Z) ≤ I(X;Y). Processing cannot increase information. A compression that loses information can never be recovered by subsequent processing. This is why representation bottlenecks matter and why the information retained at each layer of a neural network is a meaningful quantity.
| KL divergence D_KL(P | Q) is not a metric — it is asymmetric and does not satisfy the triangle inequality — but it has a clean interpretation: it is the expected number of extra bits required to encode samples from P using a code optimized for Q, rather than the optimal code for P. Forward KL (minimizing D_KL(P | Q) over Q) produces mode-covering behavior: Q tries to have support wherever P has support. Reverse KL (minimizing D_KL(Q | P) over Q) produces mode-seeking behavior: Q concentrates on high-probability regions of P. This asymmetry explains why variational inference (which minimizes reverse KL) tends to produce underestimating posterior approximations, while other training objectives produce different failure modes. |
| The Jensen-Shannon divergence JSD(P | Q) = ½D_KL(P | M) + ½D_KL(Q | M) where M = ½(P+Q) is symmetric and bounded: JSD(P | Q) ∈ [0, 1] (in bits). It is the information-theoretic quantity that generative adversarial networks (GANs) were shown to minimize — the GAN training objective is equivalent to minimizing JSD between the real data distribution and the generator distribution, a connection established by Goodfellow et al. in 2014 that illuminated why GANs have the training dynamics they do. |
Source Coding: Compression and Its Limits
The source coding theorem establishes that H(P) bits per symbol is the fundamental limit of lossless compression. Two regimes matter in practice.
For lossless compression, codes must exactly reconstruct the original. The Kraft inequality establishes that prefix-free codes (where no codeword is a prefix of another, enabling unambiguous decoding) must satisfy Σ 2^{-l_i} ≤ 1 where l_i are codeword lengths. The optimal prefix-free code minimizes the expected length Σ p_i l_i subject to the Kraft inequality; Huffman coding achieves this optimum. For longer sequences, arithmetic coding achieves the entropy rate minus O(1) bits per symbol, approaching the Shannon limit.
For lossy compression, the rate-distortion theorem (Shannon, 1959) establishes the tradeoff between compression rate and reconstruction distortion. For a source with entropy H(P) and a distortion measure d(x, x̂), there is a rate-distortion function R(D) — the minimum rate needed to achieve expected distortion ≤ D. JPEG and MP3 are practical lossy compression schemes that implicitly optimize tradeoffs near the rate-distortion curve for image and audio sources.
The practical significance for CS is that understanding the entropy of data sources reveals what compression is possible and what is not. Text in natural language has an entropy rate of approximately 1-2 bits per character (Shannon estimated this by experiment in 1951); modern language models have approached this limit in their perplexity measurements. DNA sequences have entropy rates close to their apparent information content when dependencies are accounted for. Understanding the entropy of your data constrains what any model can achieve.
Channel Coding: Reliable Communication and Its Limits
| A channel is characterized by a conditional distribution P(Y | X) — the output Y given input X, where X is chosen from an input alphabet and Y is the received output. The channel capacity is C = max_{P(X)} I(X;Y) bits per channel use, maximized over all input distributions P(X). |
The noisy channel coding theorem says that rates R < C are achievable (arbitrarily small error probability possible) and rates R > C are not (error probability bounded away from zero). The theorem is proved in two parts. The direct coding theorem uses the probabilistic method: randomly chosen codebooks achieve capacity with high probability. The converse shows that no coding scheme can exceed capacity; Fano’s inequality is the main tool, bounding the probability of decoding error below any rate exceeding capacity.
For the binary symmetric channel (BSC) with crossover probability p, capacity is C = 1 - H_b(p) = 1 - (-p log p - (1-p) log(1-p)) bits per channel use. At p = 0.5 (completely random output), capacity is 0. At p = 0 or p = 1 (deterministic), capacity is 1. At p = 0.1, capacity is approximately 0.53 bits per channel use — so even a channel with 10% bit errors can reliably transmit at over half its nominal rate with appropriate coding.
The AWGN (additive white Gaussian noise) channel, the continuous-alphabet channel that models wireless communication, has capacity C = ½ log₂(1 + SNR) bits per channel use where SNR is the signal-to-noise ratio. This is the Shannon-Hartley theorem. At SNR = 1 (0 dB), capacity is 0.5 bits per channel use; at SNR = 10 (10 dB), capacity is approximately 1.73 bits per channel use. The theorem explains the fundamental limits of wireless communication regardless of modulation scheme or coding.
What Studying This Changes
Information theory changes how practitioners understand compression, communication, and learning as related activities.
The first change is being able to read the fundamental limits. A practitioner who understands entropy and channel capacity knows when a compression or transmission problem is near its limit and when there is room for improvement. If a compressor is achieving 3 bits per symbol on a source with entropy 1 bit per symbol, the source model is wrong or the code is inefficient — either way, the gap is computable and actionable. If a neural network’s test perplexity equals its training perplexity and approaches the entropy rate of the language, further improvement requires better data or different objectives, not architecture search.
The second change is the interpretation of machine learning objectives. Cross-entropy loss is KL divergence. The ELBO is reconstruction minus KL. Contrastive learning maximizes mutual information. These are not just names — they are descriptions of what the training objective is actually optimizing, what success looks like, and what the fundamental limits are. A practitioner who understands this can diagnose training failures more precisely (high KL divergence term in VAE training means the approximate posterior is not matching the prior — why?) and can evaluate proposed training objectives by their information-theoretic properties.
The third change is the vocabulary for discussing representations. The data processing inequality constrains what any representation can capture: if you compress information to produce a representation, you cannot recover what was lost by subsequent processing. The mutual information between a representation and the target variable is what matters for prediction. The information bottleneck principle suggests that good representations compress the input while preserving task-relevant information. These ideas give practitioners a language for thinking about representation quality that goes beyond empirical evaluation.
The fourth change is the connection to coding theory and reliable systems. Error-correcting codes are information theory applied to communication and storage. A practitioner who understands channel capacity and the Shannon-Hartley theorem understands why 5G wireless works the way it does, why QR codes have redundancy, and why RAID storage can recover from disk failure. These are not academic results — they are the mathematical foundation of the communication and storage infrastructure that all software runs on.
Resources
Books and Texts
Cover and Thomas’s Elements of Information Theory (2nd ed., Wiley, 2006) is the canonical mathematical reference. It covers entropy, mutual information, the source and channel coding theorems, rate-distortion theory, Kolmogorov complexity, and network information theory with full proofs and careful attention to technical conditions. It is rigorous and comprehensive — the right primary reference for anyone who wants the full mathematical treatment. The notation and proof style are standard and will be recognized throughout the information theory literature.
MacKay’s Information Theory, Inference, and Learning Algorithms (Cambridge, 2003, free at inference.org.uk) is the alternative with a different emphasis. MacKay develops information theory in connection with Bayesian inference, error-correcting codes, and machine learning — making the connections to statistical learning explicit rather than treating information theory as a separate subject. The writing is unusually clear and engaging for a technical textbook, and the book’s integration of information theory with inference and learning makes it the right primary text for machine learning practitioners. It is both broader (covering neural networks, Monte Carlo methods, and approximate inference) and less technically complete than Cover-Thomas. The free availability and the ML focus make it the more commonly recommended starting point for CS learners.
Shannon’s original 1948 paper “A Mathematical Theory of Communication” (Bell System Technical Journal) is short (79 pages), readable, and historically central. The entropy definition, the source coding theorem, the channel capacity formula, and the noisy channel coding theorem are all in this single paper. Reading it after studying an introductory text conveys something about how the ideas emerged that no secondary account captures. It is freely available through Bell Labs archives and various academic repositories.
For Kolmogorov complexity and algorithmic information theory, Li and Vitányi’s An Introduction to Kolmogorov Complexity and Its Applications (4th ed., Springer, 2019) is the standard reference. It covers Kolmogorov complexity, the incompressibility method, MDL, and the connections to computability and complexity theory with full rigor.
For error-correcting codes specifically, Blahut’s Algebraic Codes for Data Transmission (Cambridge, 2003) covers algebraic coding theory at depth, including Reed-Solomon, BCH, and turbo codes. Richardson and Urbanke’s Modern Coding Theory (Cambridge, 2008) covers LDPC and turbo codes with the probabilistic methods that led to capacity-achieving codes.
Moser and Chen’s A Student’s Guide to Coding and Information Theory (Cambridge, paid) is a short, accessible bridge from practical coding examples to entropy, mutual information, channel capacity, and the information transmission theorem. It is much less deep than Cover-Thomas and less ML-oriented than MacKay, but it is useful for learners who need the communication-and-coding intuition before the full mathematical treatment.
| Book | Role | Type |
|---|---|---|
| Cover & Thomas, Elements of Information Theory (2nd ed.) | Canonical mathematical reference | Reference |
| MacKay, Information Theory, Inference, and Learning Algorithms (free) | ML-oriented entry; unified treatment | Entry |
| Li & Vitányi, Introduction to Kolmogorov Complexity (4th ed.) | Algorithmic information theory | Depth |
| Richardson & Urbanke, Modern Coding Theory | LDPC and turbo codes | Depth |
| Shannon, “A Mathematical Theory of Communication” (1948, free) | Founding paper; historically central | Auxiliary |
| Moser & Chen, A Student’s Guide to Coding and Information Theory (paid) | Gentle coding-first bridge | Auxiliary |
Courses and Lectures
MIT 6.441 (Information Theory, free on MIT OCW) is the standard graduate course using Cover-Thomas. Full lecture notes, problem sets, and exams are available. The course emphasizes mathematical rigor and covers the standard topics of the Cover-Thomas curriculum.
David MacKay’s Cambridge lectures on Information Theory (free on YouTube from his 2012 course) accompany his textbook and cover both information theory and its connections to machine learning and Bayesian inference. MacKay’s teaching is unusually clear and his explanations of the entropy axioms, the source coding theorem, and the connections to ML are the most illuminating available.
Stanford EE376A (Information Theory, materials from Tsachy Weissman and others, available online) covers the course in a more applied direction with connections to statistics and data compression.
| Course | Platform | Type |
|---|---|---|
| MIT 6.441 Information Theory (free) | MIT OCW | Depth |
| MacKay’s Cambridge Information Theory lectures (free) | YouTube | Entry |
| Stanford EE376A Information Theory (free) | Stanford course site | Depth |
Practice, Tools, and Current Sources
| Computing entropy and mutual information for concrete distributions is the primary way to build information-theoretic intuition. A productive sequence of calculations: compute the entropy of a fair coin, a biased coin, a fair die, and a uniform distribution over 256 characters; observe how entropy scales with the log of the number of equally likely outcomes; observe that H(X) ≤ log | X | with equality iff X is uniform. Then compute the entropy of English text empirically (by counting letter frequencies in a sample), and compare to the Shannon estimate of 1-2 bits per character — which requires looking at longer contexts than individual letters. |
For mutual information in practice: compute I(X;Y) for two correlated Gaussian variables with known correlation ρ, observing how mutual information relates to correlation but is not the same. Compute the mutual information between features and labels in a simple dataset using kernel density estimation or histogram methods, and observe the computational difficulties — mutual information estimation is hard in high dimensions, which explains why contrastive learning methods use proxy objectives.
For compression: implement Huffman coding from scratch for a given symbol distribution, compute the expected code length, and compare to the entropy. Then compare to arithmetic coding’s performance on the same distribution. Observe that for very skewed distributions (one symbol with probability close to 1), Huffman wastes a bit per symbol that arithmetic coding avoids. Nayuki’s reference implementations of Huffman coding and arithmetic coding (free, MIT-licensed) are useful for checking one’s own implementation against clear multi-language code optimized for readability rather than performance tricks.
For channel coding: implement a simple Hamming (7,4) code, introduce random bit errors at various rates, and measure the decoded error rate. Plot against the BSC channel capacity and observe where the Hamming code’s error rate crosses the theoretical limits.
SciPy (scipy.stats.entropy) and scikit-learn (sklearn.metrics.mutual_info_score) provide implementations of entropy and mutual information for empirical distributions. PyTorch and TensorFlow implement cross-entropy loss; understanding that these are information-theoretic quantities rather than arbitrary loss functions is the practical payoff of studying information theory.
| Resource | Platform | Type |
|---|---|---|
| Entropy and mutual information calculations | Python / SciPy | Practice |
| Huffman coding implementation | Local | Practice |
| Nayuki reference implementations: Huffman and arithmetic coding (free) | GitLab / nayuki.io | Practice |
| Hamming code implementation | Local | Practice |
| SciPy entropy and MI functions (free) | scipy.org | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Treating cross-entropy loss as an arbitrary choice | Many ML practitioners use cross-entropy loss because it works, without understanding that it is the negative log-likelihood under the model distribution — equivalently, the KL divergence from data distribution to model distribution plus a constant. This makes cross-entropy seem like one of several options rather than the principled information-theoretic training objective it is. | Derive cross-entropy loss from first principles as maximum likelihood estimation: write the log-likelihood of the training data under the categorical model, take the negative, observe that it equals cross-entropy. Then observe that minimizing cross-entropy is minimizing KL divergence from data distribution to model. The derivation takes fifteen minutes and makes all subsequent encounters with the loss function interpretable. |
| Confusing entropy with KL divergence with cross-entropy | These are three different quantities that are closely related and frequently conflated. Entropy H(P) measures uncertainty in a distribution. KL divergence D_KL(P||Q) measures how much P differs from Q, asymmetrically. Cross-entropy H(P,Q) = H(P) + D_KL(P||Q) measures the expected code length under model Q for data from P. Conflating them produces wrong intuitions about what training objectives optimize. | Learn all three explicitly with concrete examples before using any of them. Compute entropy, KL divergence, and cross-entropy for two specific distributions (e.g., a fair coin and a biased coin with probability 0.7) and verify the algebraic relationships among them. |
| Applying continuous information theory without understanding the measure-theoretic subtleties | Differential entropy h(X) for continuous variables can be negative (unlike discrete entropy), is not invariant under reparameterization, and cannot be directly compared to discrete entropy. Many presentations of information theory smooth over these differences, leading to incorrect conclusions when continuous distributions are analyzed. | Read the relevant sections of Cover-Thomas on differential entropy carefully, particularly the discussion of how differential entropy changes under transformation. When computing mutual information for continuous variables, understand that only mutual information (not differential entropy alone) is invariant under reparameterization. |
| Skipping the source and channel coding theorems as “just communications” | The source coding theorem and channel coding theorem are the most important results in information theory, but learners with ML focus sometimes skip them because they seem unrelated to learning. This misses that the fundamental limits they establish apply to any information processing — including learning — and that the proofs introduce the probabilistic method, random coding arguments, and the concept of typicality that reappear throughout ML theory. | Read at least the statement and proof sketch of both theorems in MacKay’s book, even if you never work with communication systems. The random coding argument that proves the channel coding theorem is one of the most elegant applications of the probabilistic method and is worth understanding for that reason alone. |
| Missing the connection to minimum description length | Minimum description length (MDL) is the statistical model selection principle that chooses the model that most compresses the data — the shortest description of both the model and the data given the model. MDL provides an information-theoretic foundation for regularization and for Bayesian model selection, connecting information theory to statistics in a way that neither field makes clear on its own. | Read MacKay’s chapters on MDL and on the relationship between compression and inference. Then ask: what does L1 regularization correspond to in MDL terms? (A Laplacian prior on weights.) What does early stopping correspond to? (Implicit model complexity limitation.) These connections make regularization techniques interpretable. |
| Treating mutual information as easy to compute | Mutual information is theoretically clean but computationally difficult for high-dimensional continuous data. Histogram estimators are biased; kernel density estimators scale poorly; most practical MI estimators have significant variance. Many ML papers claiming to maximize or minimize mutual information are using proxy objectives that may not closely track actual MI. | Before accepting a “mutual information maximization” claim, ask: how is MI computed? If the answer involves a neural estimator or a lower bound, understand the gap between the estimated quantity and the true MI. The literature on MI estimation (MINE, InfoNCE, DV estimators) discusses these gaps explicitly. |
中文
信息是解除不确定性的东西。当你收到一条消息时，它所包含的信息，正是它减少你对世界不确定性的量。这个定义听起来像是循环定义，直到你追问：不确定性可以被测量吗？Claude Shannon 在 1948 年一篇奠定整个学科的论文中给出的答案是：可以。而且这个度量具有一种特定形式，它必然来自少数几个关于“不确定性度量应当是什么样子”的约束。
概率分布 P 在各个结果上的熵是 H(P) = -Σ p(x) log₂ p(x)，单位是 bit。一枚公平硬币的熵是 1 bit：需要一个 yes/no 问题才能确定结果。一枚公平骰子的熵是 log₂ 6 ≈ 2.58 bits。确定性结果的熵是 0 bit：没有需要解除的不确定性。这不是一个约定，而是一个定理：在相差一个正乘法常数的意义下，熵是唯一满足关于不确定性度量的自然公理的函数。这个定义捕捉了某种真实内容。
对计算机科学来说，信息论提供三样东西。第一，是描述不确定性、冗余和压缩的定量语言——每一种无损压缩算法都受到其信息源熵率的限制，而高效算法会接近这个界限。第二，是推理有噪声信道通信的数学基础——Shannon 的信道容量定理告诉你，无论使用什么编码方案，信息能够被可靠传输的最大速率是多少。第三，是机器学习的统一词汇——cross-entropy loss 是信息论的，KL divergence 是信息论的，variational autoencoders 中的 evidence lower bound 是信息论的，mutual information 支撑许多现代表征学习目标。理解信息论，会让机器学习的数学变得可解释，而不是不透明。
前置知识：概率与统计（§2.5）——entropy、KL divergence 和 mutual information 都以概率分布定义，而信源编码定理和信道编码定理都是概率性结果。微积分（§2.4）用于连续情形。
信息概念如何变得精确
1948 年之前，“信息”只是一个日常词，没有数学意义。从事电话和电报系统的工程师知道信号携带信息，也知道噪声会损害信息，但信息概念本身仍然含糊——无法测量，无法比较，也无法优化。Claude Shannon 1948 年发表在 Bell System Technical Journal 上的论文 “A Mathematical Theory of Communication”，用单篇论文改变了这一点；它是科学史上最具后果的论文之一。
Shannon 的出发问题是实践性的：在不丢失内容的情况下，一条消息最多能被压缩到什么程度？在有噪声信道上，信息最多能以多快速度被可靠传输？这些问题原本看起来像需要工程答案的工程问题。Shannon 表明，它们是具有数学答案的数学问题——只要统计性质被指定，这些答案就不依赖具体编码、具体信号或具体噪声。
数学先驱是分散且局部的。Harry Nyquist 在 1928 年推导出采样定理，确立了带限信号可以由两倍带宽频率的离散采样完美重建——如今这一结果被理解为 Shannon 框架的一个特例。Ralph Hartley 在 1928 年提出过信息容量的对数度量。Norbert Wiener 在 MIT 独立研究控制论和信号处理时，也在发展类似思想；同一时期 Shannon 正在 Bell Labs 工作。Shannon 了解并建立在这些先前工作之上，但他的综合——尤其是信道容量定理和有噪声信道编码定理——远远超过了前人达到的范围。
熵的定义来自唯一性论证。Shannon 问：不确定性的度量应当具有什么性质？它应当关于概率连续。它应当在所有结果等可能时达到最大。它还应当满足一个一致性条件：联合结果的不确定性应当等于第一个结果的不确定性，加上在已知第一个结果后剩余不确定性的期望值。这些公理在相差一个常数的意义下唯一确定了熵（Shannon，1948；Khinchin 在 1957 年给出了更严格证明）。对数底数决定单位：以 2 为底得到 bits，以 e 为底得到 nats，以 10 为底得到 dits。选择哪种单位是约定；这个函数本身不是约定。
信源编码定理确立了无损压缩的根本极限。给定一个信源，它会从分布 P 中独立抽取符号；那么没有任何无损压缩方案能把该信源平均压缩到低于 H(P) bits 每符号——同时也存在能够任意接近这一界限的方案。Shannon 证明的编码方案是构造性的，但并不高效；实用方案后来才出现。David Huffman 当时是 MIT 研究生，他在 1952 年把一个课程作业做成了最优 prefix-free code；此前他被告知，如果能找到比既有方法更好的构造，就可以免期末考试。Huffman algorithm 对逐符号编码是最优的；arithmetic coding 在 1970 年代和 1980 年代发展出来，能够对更长序列接近熵率，并支撑现代无损压缩器，包括 gzip 和 PNG 使用的 DEFLATE 格式。
有噪声信道编码定理是更深、也更令人意外的结果。有噪声信道——即传输符号在抵达前可能被破坏的信道——具有容量 C，单位是 bits per channel use。如果以速率 R < C 传输，就存在能够使错误概率任意小的编码方案。如果以速率 R > C 传输，那么无论使用什么编码方案，错误都不可避免。这个定理是存在性证明：Shannon 通过概率论证说明这样的代码必然存在——随机选择的代码会以高概率达到界限——但没有构造它们。寻找能够接近信道容量的实用代码又花了几十年。Hamming codes（1950）是最早的代数纠错码。Reed-Solomon codes（1960）被用于 CD、DVD 和太空通信。Turbo codes（1993）以及从 Gallager 1962 年博士论文中在 1990 年代重新发现的 LDPC codes，已经接近 Shannon limit 到不足一分贝的差距。Polar codes（Arıkan，2009）是第一类被证明可以对任意对称二进制输入信道达到信道容量的代码，并被用于 5G 通信。
| 信息论框架通过与统计和概率的汇合，扩展到原本的通信语境之外。Solomon Kullback 和 Richard Leibler 在 1951 年形式化了如今称为 KL divergence 的量：D_KL(P | Q) = Σ p(x) log(p(x)/q(x))，它测量从分布 Q 更新到分布 P 时获得的信息量。KL divergence 出现在统计推断的各个地方：maximum likelihood estimator 最小化从数据分布到模型的 KL divergence；Bayesian updating 可以理解为最小化从 prior 到 posterior 的 KL divergence；maximum entropy principle（Jaynes，1957）会在约束下选择熵最大的分布，并由此证明 exponential family 是具有充分统计量的分布类。 |
Andrey Kolmogorov、Gregory Chaitin 和 Ray Solomonoff 在 1960 年代分别独立发展出 algorithmic information theory——一种衡量单个字符串而不是分布的复杂性度量。字符串 x 的 Kolmogorov complexity K(x)，是能输出 x 的最短程序长度。不同于 Shannon entropy，后者是分布的性质；Kolmogorov complexity 是单个字符串的性质。但二者深刻相连：如果随机变量 X 来自分布 P，那么其期望 Kolmogorov complexity 近似等于 Shannon entropy H(P)。Algorithmic information theory 为 minimum description length（MDL）提供基础；MDL 是一种模型选择原则，在模型复杂度和拟合优度之间权衡，也照亮了压缩与统计推断之间的关系。
| 当代时期，信息论已经成为机器学习的基础语言。Cross-entropy loss 用于训练几乎所有分类模型，它是正确标签期望 log-probability 的负值：H(P, Q) = -Σ p(x) log q(x)，也就是数据分布 P 与模型 Q 之间的 cross-entropy。这不只是一个任意 loss function——它等于从 P 到 Q 的 KL divergence 加上 P 的 entropy；最小化它等价于最小化 KL divergence，也等价于 maximum likelihood estimation。Variational autoencoders 中的 evidence lower bound（ELBO）可以分解为 reconstruction term 减去 KL divergence，使 VAE 训练目标明确成为信息论目标。Mutual information I(X;Y) = H(X) - H(X | Y) = H(Y) - H(Y | X) 已经成为 self-supervised representation learning 方法的常用目标，这些方法包括 mutual information maximization、InfoNCE 和 contrastive learning variants，它们在没有标签的情况下学习表征。Information bottleneck method（Tishby、Pereira、Bialek，1999；Tishby 和 Schwartz-Ziv 在 2017 年重新考察）提出，深度学习通过压缩输入信息、同时保留任务相关信息来工作——这一框架引发了大量争论，争论焦点是它是否准确描述了神经网络实际做的事。无论这个具体主张是否正确，用信息论词汇讨论表征中包含什么，已经成为标准做法。 |
熵、信道与分布的几何
熵及其扩展
熵 H(X) = -Σ p(x) log p(x) 是基本量，但为了推理多个变量以及分布之间的差异，还需要若干扩展。
| 联合熵 H(X, Y) = -Σ_{x,y} p(x,y) log p(x,y) 测量一对变量中的总不确定性。条件熵 H(X | Y) = H(X,Y) - H(Y) 测量观察到 Y 之后，X 中剩余的不确定性。链式法则 H(X,Y) = H(X) + H(Y | X) 把联合熵分解为边缘项加条件项，并可直接推广到 n 个变量。 |
| 互信息 I(X;Y) = H(X) - H(X | Y) = H(Y) - H(Y | X) = H(X) + H(Y) - H(X,Y)，测量观察 Y 后对 X 不确定性的减少，也对称地测量观察 X 后对 Y 不确定性的减少。它始终非负，且当且仅当 X 和 Y 独立时等于零。它不是 metric，因为三角不等式不成立；但它是对称的，并且不同于相关系数，它能捕捉非线性依赖。对于连续变量，微分熵 h(X) = -∫ p(x) log p(x) dx 扩展了定义，但会失去离散熵的一些性质——微分熵可以为负，也不在重参数化下保持不变。 |
数据处理不等式是信息论中最有实践价值的后果之一：如果 X → Y → Z 是 Markov chain，也就是在给定 Y 后 Z 与 X 条件独立，那么 I(X;Z) ≤ I(X;Y)。处理不能增加信息。一个丢失信息的压缩，永远无法通过后续处理恢复。这解释了为什么 representation bottlenecks 重要，也解释了为什么神经网络每一层保留的信息量是有意义的量。
| KL divergence D_KL(P | Q) 不是 metric——它不对称，也不满足三角不等式——但它有清晰解释：用针对 Q 优化的编码来编码来自 P 的样本，而不是使用针对 P 的最优编码时，平均额外需要的 bit 数。Forward KL（在 Q 上最小化 D_KL(P | Q)）会产生 mode-covering 行为：Q 会试图在 P 有支持集的地方都有支持。Reverse KL（在 Q 上最小化 D_KL(Q | P)）会产生 mode-seeking 行为：Q 会集中到 P 的高概率区域。这个不对称性解释了为什么 variational inference，也就是最小化 reverse KL，倾向于产生低估 posterior 的近似；而其他训练目标会产生不同失败模式。 |
| Jensen-Shannon divergence JSD(P | Q) = ½D_KL(P | M) + ½D_KL(Q | M)，其中 M = ½(P+Q)，是对称且有界的：JSD(P | Q) ∈ [0, 1]（以 bits 为单位）。它是 generative adversarial networks（GANs）被证明所最小化的信息论量——GAN 训练目标等价于最小化真实数据分布和生成器分布之间的 JSD。这一联系由 Goodfellow 等人在 2014 年确立，并照亮了为什么 GAN 具有它们那样的训练动态。 |
信源编码：压缩及其极限
信源编码定理确立了 H(P) bits per symbol 是无损压缩的根本极限。实践中有两个区域很重要。
对于无损压缩，编码必须精确重建原始内容。Kraft inequality 确立了 prefix-free codes，也就是没有任何码字是另一个码字前缀、从而能够无歧义解码的代码，必须满足 Σ 2^{-l_i} ≤ 1，其中 l_i 是码字长度。最优 prefix-free code 在 Kraft inequality 约束下最小化期望长度 Σ p_i l_i；Huffman coding 达到这一最优。对于更长序列，arithmetic coding 可以达到熵率减 O(1) bits per symbol，接近 Shannon limit。
对于有损压缩，rate-distortion theorem（Shannon，1959）确立了压缩率与重建失真之间的取舍。对于熵为 H(P) 的信源以及失真度量 d(x, x̂)，存在一个 rate-distortion function R(D)——它是在期望失真 ≤ D 时所需的最小速率。JPEG 和 MP3 是实际有损压缩方案，它们隐式地为图像和音频信源优化接近 rate-distortion curve 的取舍。
对 CS 的实践意义在于，理解数据信源的熵会揭示什么压缩是可能的、什么是不可能的。自然语言文本的熵率约为每字符 1–2 bits（Shannon 在 1951 年通过实验估计了这一点）；现代语言模型在 perplexity 测量上已经接近这一极限。DNA 序列在考虑依赖关系后，其熵率接近其表观信息内容。理解数据的熵，会约束任何模型能够达到的结果。
信道编码：可靠通信及其极限
| 一个信道由条件分布 P(Y | X) 刻画——在给定输入 X 时得到输出 Y，其中 X 从输入字母表中选择，Y 是接收到的输出。信道容量是 C = max_{P(X)} I(X;Y) bits per channel use，在所有输入分布 P(X) 上取最大值。 |
有噪声信道编码定理说，速率 R < C 是可达到的，也就是可以获得任意小的错误概率；速率 R > C 是不可达到的，也就是错误概率会与零保持距离。这个定理分两部分证明。直接编码定理使用概率方法：随机选择的码本会以高概率达到容量。逆定理表明，没有任何编码方案能超过容量；Fano’s inequality 是主要工具，它会对超过容量时的解码错误概率给出下界。
对于 crossover probability 为 p 的 binary symmetric channel（BSC），容量为 C = 1 - H_b(p) = 1 - (-p log p - (1-p) log(1-p)) bits per channel use。当 p = 0.5 时，输出完全随机，容量为 0。当 p = 0 或 p = 1 时，信道是确定性的，容量为 1。当 p = 0.1 时，容量约为 0.53 bits per channel use——因此，即使一个信道有 10% bit errors，只要使用合适编码，也能以超过其名义速率一半的速度可靠传输。
AWGN（additive white Gaussian noise）信道是用于建模无线通信的连续字母表信道，其容量为 C = ½ log₂(1 + SNR) bits per channel use，其中 SNR 是 signal-to-noise ratio。这就是 Shannon-Hartley theorem。当 SNR = 1（0 dB）时，容量为 0.5 bits per channel use；当 SNR = 10（10 dB）时，容量约为 1.73 bits per channel use。该定理解释了无线通信的根本极限，而不依赖调制方案或编码方式。
学习这个主题会改变什么
信息论会改变实践者理解压缩、通信和学习这些活动之间关系的方式。
第一种变化，是能够读出根本极限。理解熵和信道容量的实践者，知道一个压缩或传输问题什么时候接近极限，什么时候还有改进空间。如果一个压缩器在熵为 1 bit per symbol 的信源上达到 3 bits per symbol，那么要么信源模型是错的，要么编码效率低；无论哪种，差距都是可计算且可行动的。如果一个神经网络的测试 perplexity 等于训练 perplexity，并且接近语言的熵率，那么进一步改进需要更好的数据或不同目标，而不是架构搜索。
第二种变化，是能解释机器学习目标。Cross-entropy loss 是 KL divergence。ELBO 是 reconstruction 减去 KL。Contrastive learning 最大化 mutual information。这些不只是名称——它们描述了训练目标实际在优化什么，成功意味着什么，以及根本极限在哪里。理解这一点的实践者，可以更精确地诊断训练失败，例如 VAE 训练中 KL divergence 项过高意味着近似 posterior 没有匹配 prior——为什么？也可以根据信息论性质评估新提出的训练目标。
第三种变化，是拥有讨论表征的词汇。数据处理不等式约束了任何表征能够捕捉的东西：如果你压缩信息以产生表征，随后处理无法恢复已经丢失的信息。表征与目标变量之间的 mutual information，才是预测真正关心的内容。Information bottleneck principle 表明，好的表征会压缩输入，同时保留任务相关信息。这些思想给实践者一种思考表征质量的语言，不只是依赖经验评估。
第四种变化，是连接到编码理论和可靠系统。纠错码是信息论应用于通信和存储。理解信道容量和 Shannon-Hartley theorem 的实践者，会理解为什么 5G wireless 以这种方式工作，为什么 QR codes 有冗余，为什么 RAID storage 可以从磁盘故障中恢复。这些不是学院式结果——它们是所有软件运行其上的通信与存储基础设施的数学基础。
资源
书籍与文本
Cover 和 Thomas 的 Elements of Information Theory（第 2 版，Wiley，2006）是经典数学参考。它覆盖熵、互信息、信源和信道编码定理、rate-distortion theory、Kolmogorov complexity，以及 network information theory，并给出完整证明，同时仔细处理技术条件。它严格而综合——对于任何想获得完整数学处理的人，这是正确主参考。其记号和证明风格是标准的，在整个信息论文献中都能被识别。
MacKay 的 Information Theory, Inference, and Learning Algorithms（Cambridge，2003，可在 inference.org.uk 免费获取）是另一种重点不同的选择。MacKay 把信息论与 Bayesian inference、纠错码和机器学习联系起来发展——它明确连接统计学习，而不是把信息论当成孤立主题。对一本技术教材来说，它的写作异常清晰且有吸引力；它对信息论、推断和学习的整合，使其成为机器学习实践者的正确主文本。它比 Cover-Thomas 更广，覆盖神经网络、Monte Carlo methods 和 approximate inference；但技术上没有 Cover-Thomas 完整。免费获取和 ML 取向，使它成为 CS 学习者更常被推荐的起点。
Shannon 1948 年的原始论文 “A Mathematical Theory of Communication”（Bell System Technical Journal）篇幅短，79 页，可读，并且具有历史中心性。熵定义、信源编码定理、信道容量公式和有噪声信道编码定理，全都在这一篇论文中。学习过导论文本后阅读它，可以获得任何二手说明都无法捕捉的思想发生感。它可通过 Bell Labs archives 和各种学术仓库免费获取。
对于 Kolmogorov complexity 和 algorithmic information theory，Li 和 Vitányi 的 An Introduction to Kolmogorov Complexity and Its Applications（第 4 版，Springer，2019）是标准参考。它以完整严格性覆盖 Kolmogorov complexity、incompressibility method、MDL，以及与可计算性和复杂性理论的联系。
专门针对 纠错码，Blahut 的 Algebraic Codes for Data Transmission（Cambridge，2003）深入覆盖代数编码理论，包括 Reed-Solomon、BCH 和 turbo codes。Richardson 和 Urbanke 的 Modern Coding Theory（Cambridge，2008）覆盖 LDPC 和 turbo codes，并使用通向 capacity-achieving codes 的概率方法。
Moser 和 Chen 的 A Student’s Guide to Coding and Information Theory（Cambridge，付费）是一座短而易进入的桥梁，从实际编码例子通向熵、互信息、信道容量和信息传输定理。它远不如 Cover-Thomas 深，也不如 MacKay 面向 ML，但适合那些在进入完整数学处理前，需要先建立通信与编码直觉的学习者。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Cover & Thomas，Elements of Information Theory（第 2 版） | 经典数学参考 | 参考 |
| MacKay，Information Theory, Inference, and Learning Algorithms（免费） | 面向 ML 的入口；统一处理 | 入门 |
| Li & Vitányi，Introduction to Kolmogorov Complexity（第 4 版） | 算法信息论 | 深入 |
| Richardson & Urbanke，Modern Coding Theory | LDPC 和 turbo codes | 深入 |
| Shannon，“A Mathematical Theory of Communication”（1948，免费） | 奠基论文；历史中心 | 辅助 |
| Moser & Chen，A Student’s Guide to Coding and Information Theory（付费） | 温和的编码优先桥梁 | 辅助 |
课程与讲座
MIT 6.441（Information Theory，MIT OCW 免费）是标准研究生课程，使用 Cover-Thomas。完整讲义、习题和考试都可获取。课程强调数学严格性，覆盖 Cover-Thomas 课程体系的标准主题。
David MacKay 的 Cambridge 信息论讲座（其 2012 年课程可在 YouTube 免费获取）配套他的教材，覆盖信息论及其与机器学习、Bayesian inference 的连接。MacKay 的教学异常清楚；他对熵公理、信源编码定理以及与 ML 连接的解释，是现有材料中最有启发性的。
Stanford EE376A（Information Theory，Tsachy Weissman 等人的材料可在线获取）以更应用的方向覆盖课程，并连接统计和数据压缩。
| 课程 | 平台 | 类型 |
|---|---|---|
| MIT 6.441 Information Theory（免费） | MIT OCW | 深入 |
| MacKay’s Cambridge Information Theory lectures（免费） | YouTube | 入门 |
| Stanford EE376A Information Theory（免费） | Stanford 课程网站 | 深入 |
实践、工具与当前来源
| 为具体分布计算熵和互信息，是建立信息论直觉的主要方式。一组有效计算顺序是：计算公平硬币、偏置硬币、公平骰子，以及 256 个字符上的均匀分布的熵；观察熵如何随着等可能结果数量的对数增长；观察 H(X) ≤ log | X | ，且当且仅当 X 均匀时取等号。然后经验计算英文文本的熵，例如统计样本文本中的字母频率，并与 Shannon 估计的每字符 1–2 bits 对比——这要求观察比单个字母更长的上下文。 |
对于 实践中的 mutual information：计算两个具有已知相关系数 ρ 的相关 Gaussian 变量之间的 I(X;Y)，观察 mutual information 与 correlation 如何相关但并不相同。使用 kernel density estimation 或 histogram methods，在一个简单数据集中计算特征与标签之间的 mutual information，并观察计算困难——高维 mutual information estimation 很难，这解释了为什么 contrastive learning 方法会使用 proxy objectives。
对于 压缩：针对给定符号分布从零实现 Huffman coding，计算期望码长，并与熵比较。再比较 arithmetic coding 在同一分布上的性能。观察对于高度偏斜分布，例如某个符号概率接近 1 时，Huffman 每个符号会浪费 arithmetic coding 可以避免的 bit。Nayuki 的 Huffman coding 和 arithmetic coding 参考实现（免费，MIT-licensed）可以用来检查自己的实现；其多语言代码清晰，强调可读性而非性能技巧。
对于 信道编码：实现一个简单的 Hamming (7,4) code，在不同速率下引入随机 bit errors，并测量解码后错误率。将其与 BSC 信道容量作图比较，并观察 Hamming code 的错误率在哪里越过理论极限。
SciPy（scipy.stats.entropy）和 scikit-learn（sklearn.metrics.mutual_info_score）提供了经验分布的熵和互信息实现。PyTorch 和 TensorFlow 实现了 cross-entropy loss；理解这些是信息论量，而不是任意 loss function，是学习信息论的实践回报。
| 资源 | 平台 | 类型 |
|---|---|---|
| 熵与互信息计算 | Python / SciPy | 实践 |
| Huffman coding 实现 | 本地 | 实践 |
| Nayuki 参考实现：Huffman 和 arithmetic coding（免费） | GitLab / nayuki.io | 实践 |
| Hamming code 实现 | 本地 | 实践 |
| SciPy entropy 和 MI 函数（免费） | scipy.org | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 | ||||
|---|---|---|---|---|---|---|
| 把 cross-entropy loss 当作任意选择 | 许多 ML 实践者使用 cross-entropy loss，是因为它有效，却不理解它是模型分布下的 negative log-likelihood——等价地，是从数据分布到模型分布的 KL divergence 加上常数。这会让 cross-entropy 看起来像若干可选项之一，而不是有原则的信息论训练目标。 | 从第一原则推导 cross-entropy loss 作为 maximum likelihood estimation：写出训练数据在 categorical model 下的 log-likelihood，取负值，观察它等于 cross-entropy。然后观察最小化 cross-entropy 就是在最小化从数据分布到模型的 KL divergence。这个推导只需十五分钟，并会让此后遇到 loss function 时都能解释其含义。 | ||||
| 混淆 entropy、KL divergence 和 cross-entropy | 这是三个彼此密切相关但不同的量，而且经常被混淆。Entropy H(P) 测量一个分布中的不确定性。KL divergence D_KL(P | Q) 非对称地测量 P 与 Q 有多不同。Cross-entropy H(P,Q) = H(P) + D_KL(P | Q)，测量用模型 Q 编码来自 P 的数据时的期望码长。混淆它们会导致对训练目标优化内容的错误直觉。 | 在使用任何一个之前，用具体例子明确学习三者。对两个具体分布计算 entropy、KL divergence 和 cross-entropy，例如公平硬币和正面概率 0.7 的偏置硬币，并验证它们之间的代数关系。 | ||
| 不理解测度论细节就应用连续信息论 | 连续变量的 differential entropy h(X) 可以为负，不像离散熵；它不在重参数化下保持不变，也不能直接与离散熵比较。许多信息论呈现会略过这些差异，导致分析连续分布时得出错误结论。 | 仔细阅读 Cover-Thomas 中关于 differential entropy 的相关部分，特别是 differential entropy 在变换下如何变化的讨论。当计算连续变量的 mutual information 时，要理解只有 mutual information，而不是单独的 differential entropy，才在重参数化下保持不变。 | ||||
| 把信源和信道编码定理当作“只是通信”而跳过 | 信源编码定理和信道编码定理是信息论中最重要的结果，但以 ML 为重点的学习者有时会因为它们看起来与学习无关而跳过。这会错过它们建立的根本极限其实适用于任何信息处理，包括学习；也会错过其证明中引入的概率方法、随机编码论证和 typicality 概念，而这些会在 ML 理论中反复出现。 | 至少在 MacKay 书中阅读这两个定理的陈述和证明草图，即使你从不处理通信系统。证明信道编码定理的随机编码论证，是概率方法最优雅的应用之一，仅凭这一点就值得理解。 | ||||
| 错过与 minimum description length 的连接 | Minimum description length（MDL）是统计模型选择原则，它选择最能压缩数据的模型——也就是模型本身和在模型下数据的最短描述。MDL 为正则化和 Bayesian model selection 提供信息论基础，以一种单独学习两个领域时不明显的方式，把信息论连接到统计。 | 阅读 MacKay 关于 MDL 以及压缩与推断关系的章节。然后追问：L1 regularization 在 MDL 意义上对应什么？对应权重上的 Laplacian prior。Early stopping 对应什么？对应隐式限制模型复杂度。这些连接会让正则化技术变得可解释。 | ||||
| 把 mutual information 当作容易计算的量 | Mutual information 在理论上干净，但对高维连续数据来说计算困难。Histogram estimators 有偏；kernel density estimators 扩展性差；多数实践 MI estimators 方差显著。许多声称最大化或最小化 mutual information 的 ML 论文，实际上使用的是可能并不紧密追踪真实 MI 的 proxy objectives。 | 在接受一个 “mutual information maximization” 主张之前，先问：MI 是怎么计算的？如果答案涉及 neural estimator 或 lower bound，就要理解估计量与真实 MI 之间的差距。MI estimation 文献（MINE、InfoNCE、DV estimators）会明确讨论这些差距。 |