English
A language model is a probability distribution over sequences of tokens. Given a sequence of preceding tokens, it assigns a probability to each possible next token. This is the entire specification of the task: predict the next token. Everything else that large language models do — answering questions, writing code, reasoning through problems, translating between languages, summarizing documents — emerges from training to predict tokens well across extraordinarily diverse text. The model that predicts next tokens well on the internet’s text has learned something about the world that produced that text.
Foundation models are large pretrained models whose learned representations transfer to a wide range of downstream tasks without task-specific training from scratch; large language models are the most visible and industrially important instance, but the category also includes multimodal, vision, speech, biological, and control models. The word “foundation” points to this generality: a single pretrained model serves as the foundation for many applications through fine-tuning, prompting, or both. This is a structural change from the preceding decade of deep learning, when every task required a model trained for that task.
What makes large language models a distinct section from §5.3 (Deep Learning) is scale and the qualitative change in behavior that scale makes practically usable. Deep learning provided the architectural substrate — the transformer — and the training methodology. LLMs added the data quantity (trillions of tokens), the model scale (billions to trillions of parameters), and capabilities that become much more reliable at scale: in-context learning, chain-of-thought prompting, code generation, instruction following, and multi-step tool use. These behaviors were not engineered as separate modules; they arose from the interaction of architecture, scale, training data, and post-training.
Prerequisites: Deep learning (§5.3) — transformers, self-attention, and training at scale. Machine learning foundations (§5.2) — evaluation methodology, especially the distinction between benchmark performance and capability. Reinforcement learning (§5.4) — RLHF draws on RL concepts throughout.
From Word Vectors to Generalist Systems
The path to large language models runs through a series of representations: from count-based statistics to learned word vectors to contextual embeddings to the transformer, each step increasing the expressive power of how language is represented.
Classical NLP before 2013 used sparse, hand-crafted feature representations — bags of words, n-gram counts, parse trees. The key limitation was that these representations treated words as independent symbols: “bank” in a financial context and “bank” in a geographical context were the same feature. This was adequate for classification tasks on short texts but inadequate for tasks that required understanding meaning across contexts.
Mikolov et al.’s word2vec (2013) demonstrated that a simple prediction task — predict a word from its neighbors, or predict neighbors from a word — trained on large corpora produced word embeddings with striking geometric properties: the vector for “king” minus “man” plus “woman” was close to the vector for “queen.” The semantic content of words was encoded in the geometry of embedding space. Word2vec worked by compression: a neural network forced to predict context from a single low-dimensional vector had to encode the distributional semantics of words efficiently. The same year, Sutskever, Vinyals, and Le demonstrated sequence-to-sequence models for machine translation, showing that encoder-decoder networks could map variable-length input sequences to variable-length output sequences.
The attention mechanism, introduced by Bahdanau, Cho, and Bengio in 2015, addressed the bottleneck in sequence-to-sequence models: compressing an entire source sequence into a single fixed-size vector lost information for long sequences. Attention allowed the decoder to selectively attend to different parts of the encoder output at each decoding step, dramatically improving machine translation on long sentences.
The transformer architecture (Vaswani et al., “Attention Is All You Need,” 2017) removed the recurrence entirely, replacing it with multi-head self-attention that allowed each position to attend to every other position simultaneously. The transformer had two major advantages over recurrent architectures: it could be trained in parallel (no sequential dependency), making it far more efficiently trainable on modern hardware, and its all-to-all attention allowed capturing long-range dependencies that recurrent architectures struggled with. The transformer was presented as a better machine translation model. Within a year, it had restructured the entire field of NLP and soon the rest of deep learning.
BERT (Devlin et al., 2018) showed the power of large-scale pretraining with bidirectional transformers. Rather than predicting the next word autoregressively (predicting left-to-right), BERT used masked language modeling: randomly mask 15% of tokens, train the model to predict the masked tokens from the surrounding context. Pretraining on a large corpus then fine-tuning on downstream tasks dramatically outperformed previous architectures on every major NLP benchmark. BERT established the pretraining-then-fine-tuning paradigm: pretrain a large model on unlabeled data, then adapt it cheaply to specific tasks.
GPT (Radford et al., 2018) explored the other direction: autoregressive pretraining, predicting each token from all preceding tokens, at larger scale than had been attempted before. GPT-2 (2019) scaled this approach to 1.5 billion parameters and showed that language models trained to predict text could generate surprisingly coherent long-form content and zero-shot transfer to downstream tasks. OpenAI briefly declined to release GPT-2’s weights citing misuse concerns — a decision that attracted both criticism and attention, foreshadowing debates that would intensify as capabilities grew.
GPT-3 (Brown et al., 2020) scaled further to 175 billion parameters and revealed something that had not been anticipated: in-context learning. Given a few examples of a task in the prompt — “Translate English to French: Sea otter → loutre de mer. Question → “ — GPT-3 could perform the translation without any parameter updates, by pattern-matching to the examples in context. This was not programmed; it emerged from scale. Models that had been adequate at next-token prediction had become something that looked more like a general-purpose problem solver. The GPT-3 paper’s impact was immediate: it launched a wave of industrial investment, research attention, and public awareness of language models that has not abated.
Kaplan et al.’s “Scaling Laws for Neural Language Models” (2020) provided the organizing principle for what followed. Language model performance improved as a smooth power law in compute, model size, and data size — without visible saturation. If the power law held, investment in scale would continue to produce proportional returns. Hoffmann et al. (“Training Compute-Optimal Large Language Models,” 2022 — the Chinchilla paper) refined this: the Kaplan laws had systematically underallocated data relative to compute, producing models that were overtrained in parameters and undertrained in data. Chinchilla, with 70 billion parameters and 1.4 trillion tokens (compared to Gopher’s 280 billion parameters and 300 billion tokens), matched or exceeded Gopher’s performance at significantly lower inference cost. The paper changed how the field allocated compute budgets and re-established that data quality and quantity mattered as much as model size.
The alignment turn came with InstructGPT (Ouyang et al., 2022). A GPT-3-scale model trained to predict internet text was capable but unreliable: it would complete prompts in ways that were plausible extensions of training data but unhelpful or harmful in context. RLHF fine-tuning addressed this by training a reward model from human preferences over model outputs, then fine-tuning the language model to maximize that reward using PPO. InstructGPT rated as substantially more helpful by human evaluators despite being smaller than base GPT-3. The technique — pretraining at scale, supervised fine-tuning, reward model training from human preferences, RL fine-tuning — became the standard pipeline for deployed language models. ChatGPT used it; GPT-4 used it; Claude used it; essentially every deployed frontier model uses it or a close variant.
The emergence of a competitive ecosystem in 2022-2023 brought several developments simultaneously. PaLM (Chowdhery et al., 2022) demonstrated that scaling to 540 billion parameters on high-quality data produced a model that broke through previous performance ceilings on several reasoning benchmarks. LLaMA (Touvron et al., Meta, 2023) showed that small models trained on more data were competitive with larger models at inference time — Chinchilla scaling applied to open-weight releases. LLaMA’s open weights catalyzed a community of researchers who could fine-tune, probe, and modify frontier-scale models, producing dozens of derivative models within months.
Direct Preference Optimization (DPO, Rafailov et al., 2023) simplified the RLHF pipeline by showing that the RL step could be replaced with a supervised objective that directly parameterized the reward as the log-ratio of policy to reference probabilities. DPO produced comparable results to RLHF with simpler training and without requiring a separate reward model or RL infrastructure.
The reasoning model era began with the release of OpenAI’s o1 in 2024, followed by o3, DeepSeek-R1, and analogous models from Anthropic and Google. These models allocate more test-time computation to hidden deliberative traces or multi-step generation before producing a final response — a form of test-time compute scaling that trades inference time for accuracy on difficult reasoning tasks. The results on mathematical and coding benchmarks were striking: problems that stumped previous state-of-the-art models became more tractable for models using extended inference. The mechanism — whether this constitutes something meaningfully called reasoning or is sophisticated pattern completion — remains actively debated.
The foundational infrastructure built around language models has matured into an ecosystem. The Hugging Face platform has made hundreds of open-weight models accessible through a common API. vLLM and TensorRT-LLM provide optimized inference. Parameter-efficient fine-tuning methods (LoRA, QLoRA) allow fine-tuning large models on consumer hardware. Retrieval-augmented generation (RAG) systems ground model outputs in retrieved documents. Agentic frameworks orchestrate multi-step interactions. What began as a research direction in 2017 is now the foundational infrastructure of an industry.
Architecture, Training, Adaptation, and Evaluation
The Transformer and Its Scaling Properties
The transformer’s core operation is multi-head self-attention. Each token representation is projected to queries, keys, and values; attention weights are computed as the softmax of scaled dot-products between queries and keys; values are aggregated using these weights. Multi-head attention runs this process independently in multiple subspaces and concatenates the results. A transformer layer stacks self-attention with a position-wise feed-forward network and layer normalization.
Autoregressive language models — GPT-style — apply self-attention with causal masking, ensuring each position can attend only to preceding positions. They are trained by cross-entropy loss on next-token prediction, which as discussed in §3.8 is equivalent to minimizing KL divergence between the model distribution and the data distribution. This means the model is trained to approximate the distribution of text in the training corpus, including all of its statistical regularities, factual content, and stylistic patterns.
Scaling properties are not arbitrary. As models grow in parameters, their performance on held-out text improves as a power law with an exponent around 0.07 per decade of parameters (Kaplan et al.). But the effective behavior changes qualitatively at certain scales: few-shot learning (GPT-3 level), reliable instruction following (InstructGPT level), complex multi-step reasoning (reasoning model level). These transitions are not smooth — they are approximately step functions in specific capability evaluations. This creates a fundamental challenge for prediction: capabilities that are absent at small scale may appear abruptly at larger scale, and capabilities visible at large scale may not be predictable from small-scale experiments.
The attention mechanism’s quadratic compute and memory cost in sequence length is the primary constraint on context window length. Flash Attention (Dao et al., 2022) and its successors compute exact attention with memory proportional to sequence length rather than its square by processing attention in blocks. Extended context windows — from 2K tokens in GPT-3 to millions of tokens in current models — have been enabled by architectural modifications (rotary position embeddings, attention variants with subquadratic complexity) and engineering improvements.
Pretraining, Fine-tuning, and Alignment
Pretraining requires data at extraordinary scale. The C4, Common Crawl, and Pile datasets used for open-weight models contain hundreds of billions to trillions of tokens. Data quality matters at this scale: deduplication (near-duplicate documents dominate the web and reduce data efficiency), quality filtering (classifiers that identify high-quality text), and composition (the mix of domains — web, code, books, scientific papers — affects what the model learns) are all active engineering decisions. Models trained on code exhibit better formal reasoning capabilities than those trained on text alone, which is why all frontier models train on substantial code.
Instruction fine-tuning takes a pretrained base model and trains it on (instruction, response) pairs — demonstrations of the model answering questions and following instructions helpfully. This changes the model’s default behavior from “continue the text” to “respond to the instruction.” The quality of instruction-following demonstrations matters: SELF-INSTRUCT (Wang et al., 2023) showed that models could generate their own instruction data by prompting larger models, enabling scaling of instruction tuning data without extensive human annotation.
RLHF adds a preference signal from human comparisons. Given two model responses to the same prompt, human annotators indicate which is better. A reward model trained on these comparisons learns a scalar reward for any response. The language model is then fine-tuned using PPO to maximize this reward while staying close to the reference policy via KL penalty. The KL penalty prevents reward hacking: without it, the language model finds responses that satisfy the reward model’s criteria while departing from sensible language. The tradeoff between reward maximization and KL penalty is the key hyperparameter of RLHF.
Known failure modes of RLHF are well-documented: sycophancy (models that agree with users to maximize approval rather than being accurate), verbosity bias (reward models prefer longer responses), and overoptimization (Goodhart’s law — as the reward model is optimized against, the model finds responses that satisfy the reward model but violate the spirit of what was rewarded). DPO’s key advantage is avoiding these failure modes partially by not requiring an explicit reward model and not using RL.
Evaluation: What Models Can and Cannot Do
Evaluation of language model capabilities is genuinely hard. Models exhibit strong performance on some benchmarks while failing on seemingly similar tasks. Performance is sensitive to prompt framing. Benchmark contamination — training data containing test items — inflates benchmark scores. And the mapping from benchmark performance to deployment capability is unreliable.
The standard benchmarks address different capability dimensions: MMLU tests knowledge across 57 subjects; HumanEval and MBPP test code generation; GSM8K and MATH test mathematical reasoning; BBH (Big-Bench Hard) tests challenging reasoning tasks; MT-Bench evaluates instruction following. Each benchmark has known failure modes: MMLU can be solved with surface pattern matching, HumanEval exercises relatively simple functions, GSM8K is largely solved by current frontier models.
Calibration — whether model confidence correlates with accuracy — is a distinct evaluation dimension from accuracy. A model that says “I’m not sure” when it is wrong and “I’m confident” when it is right is better calibrated than one that is equally confident in correct and incorrect answers. Verbalized uncertainty is not well-calibrated in current models; calibrated token probabilities are more reliable but harder to access in deployed systems.
Evaluation of deployed system behavior requires red-teaming: systematically trying to elicit harmful, incorrect, or policy-violating outputs. Red-teaming has produced a literature on failure modes — prompt injection attacks, jailbreaks, inconsistent value alignment, sycophancy — that complements benchmark evaluation. Safety evaluations include tests for dangerous capabilities (bioweapons synthesis, cyberattack assistance) that must be checked before deployment.
What Studying This Changes
Large language models change how practitioners think about what computation can do and what it cannot be trusted to do.
The first change is understanding the capability-reliability gap. Current large language models can do many things but cannot reliably guarantee correctness in high-stakes settings. The practitioner who has studied the architecture, training, and failure modes understands why: the model approximates the distribution of text in its training data, which means it produces plausible-sounding outputs that may be factually incorrect. This shapes how systems that use language models should be designed — with verification, grounding, human oversight, and graceful degradation where correctness matters.
The second change is evaluation discipline. Benchmark performance is not deployment performance. The practitioner who has studied the evaluation literature knows to test models on their specific deployment inputs, to characterize failure modes rather than just average performance, to evaluate calibration alongside accuracy, and to maintain skepticism about results that appear too good. This discipline prevents the common failure mode of deploying models on the strength of benchmark results that do not transfer.
The third change is architectural transparency. The practitioner who understands self-attention, the role of positional encodings, the function of the feed-forward layers, and how context window limits arise can interpret model behavior in terms of architectural constraints. Why does performance degrade on very long contexts? Why does the model fail to count precisely? Why do certain prompting patterns improve performance? These questions have architectural answers, and the practitioner who has studied the architecture can engage with them.
The fourth change is the ability to make calibrated deployment decisions. Which tasks justify fine-tuning versus prompting? When is RAG the right approach versus larger context? When is a 7B open-weight model appropriate versus a frontier model? What safety mitigations are necessary for which applications? These decisions require understanding the tradeoffs — capability, cost, reliability, privacy, control — and the understanding comes from study rather than from using the APIs.
Resources
Books and Texts
The foundational transformer text is now primarily papers rather than books — the field moved faster than book timescales allowed. However, Jurafsky and Martin’s Speech and Language Processing (3rd ed., free online at web.stanford.edu/~jurafsky/slp3) has been updated to cover neural language models, transformers, and large language models and remains the most complete textbook treatment. Its coverage of the classical NLP that preceded the transformer era is also valuable context.
For foundational ML and deep learning preparation, Bishop and Bishop’s Deep Learning: Foundations and Concepts (2024, §5.3 reference) and Goodfellow et al.’s Deep Learning (free, §5.3 reference) provide the necessary background.
Bommasani et al.’s “On the Opportunities and Risks of Foundation Models” (Stanford CRFM, 2021, free) is a long report rather than a book but serves as the most comprehensive single document articulating what foundation models are, what they can do, and what risks they pose. It is dated in specifics but remains the most systematic framing.
The primary literature is the authoritative source. The papers that matter most: Vaswani et al. 2017 (transformer), Devlin et al. 2018 (BERT), Brown et al. 2020 (GPT-3), Kaplan et al. 2020 (scaling laws), Hoffmann et al. 2022 (Chinchilla), Ouyang et al. 2022 (InstructGPT), Rafailov et al. 2023 (DPO). Each is 10–30 pages and directly readable by someone with the §5.3 prerequisites.
| Resource | Role | Type |
|---|---|---|
| Jurafsky & Martin, Speech and Language Processing (3rd ed., free) | LM and transformer foundations | Entry |
| Bommasani et al., “On the Opportunities and Risks of Foundation Models” (free) | Comprehensive framing report | Entry |
| Vaswani et al., “Attention Is All You Need” (2017, free) | Transformer landmark paper | Depth |
| Brown et al., GPT-3 paper (2020, free) | Few-shot learning; era-launching | Depth |
| Kaplan et al., scaling laws paper (2020, free) | Scaling relationships | Depth |
| Hoffmann et al., Chinchilla paper (2022, free) | Compute-optimal training | Depth |
| Ouyang et al., InstructGPT paper (2022, free) | RLHF alignment pipeline | Depth |
| Rafailov et al., DPO paper (2023, free) | Direct preference optimization | Depth |
| Transformer architecture reference: Phuong & Hutter (2022, free) | Precise architectural reference | Reference |
| Chip Huyen, AI Engineering (paid) | Foundation-model application engineering | Practice |
Courses and Lectures
Stanford CS336 (Language Modeling from Scratch, materials at cs336.stanford.edu) is the most technically rigorous available course on training language models, covering data engineering, architecture, training infrastructure, and evaluation. It is aimed at practitioners who want to understand how frontier models are built rather than just how to use them.
Andrej Karpathy’s Neural Networks: Zero to Hero (§5.3 reference) concludes with training a character-level GPT, building intuition for the autoregressive language model from first principles.
Hugging Face NLP Course (free at huggingface.co/course) provides practical foundation: working with tokenizers, pretrained models, fine-tuning, and the Transformers library. This is the right practical entry for practitioners building applications.
Fast.ai’s Practical Deep Learning and companion courses include sessions on language models with practical implementation focus.
| Course | Platform | Type |
|---|---|---|
| Stanford CS336 Language Modeling from Scratch | cs336.stanford.edu | Depth |
| Berkeley LLM Agents course (free) | rdi.berkeley.edu | Depth |
| Hugging Face NLP Course (free) | huggingface.co/course | Practice |
| Hugging Face Agents Course (free) | huggingface.co/learn | Practice |
| Cohere LLM University (free) | docs.cohere.com | Auxiliary |
| Karpathy LLM-adjacent modules from Neural Networks: Zero to Hero (free) | YouTube | Auxiliary |
Practice, Tools, and Current Sources
Karpathy’s nanoGPT (GitHub, free) implements a complete GPT training pipeline in approximately 300 lines of PyTorch. Training it on Shakespeare-scale or Wikipedia-scale text, then modifying the architecture or training procedure, is the most instructive hands-on engagement with language model training at accessible scale.
The Hugging Face ecosystem — Transformers, Datasets, PEFT, TRL — is the standard infrastructure for working with language models. Fluency with the Transformers API (loading models, running inference, fine-tuning with PEFT/LoRA) is a practical prerequisite for most applied work. The TRL library provides implementations of RLHF and DPO training.
LM-Evaluation-Harness (EleutherAI, free on GitHub) provides standardized evaluation of language models across dozens of benchmarks. Running a model through the harness and examining per-task performance reveals capability heterogeneity that aggregate scores obscure.
vLLM (free) provides high-throughput inference for large language models with PagedAttention, making it practical to serve larger models efficiently. Understanding how PagedAttention manages the KV cache provides insight into inference engineering.
The OpenAI API, Anthropic API, Google AI API provide access to frontier closed-weight models. Systematic comparison of model behavior across providers — on specific task types, failure modes, and prompting strategies — builds the calibrated understanding that API documentation does not.
| Resource | Platform | Type |
|---|---|---|
| Karpathy, nanoGPT (free) | GitHub | Practice |
| OpenAI Cookbook (free; API usage may be paid) | GitHub / cookbook.openai.com | Practice |
| Anthropic Cookbook (free; API usage may be paid) | GitHub | Practice |
| Prompt Engineering Guide (free) | dair.ai / promptingguide.ai | Auxiliary |
| Hugging Face Transformers / TRL (free) | huggingface.co | Practice |
| LM-Evaluation-Harness (free) | GitHub / EleutherAI | Practice |
| HELM (free) | crfm.stanford.edu/helm | Reference |
| Chatbot Arena / LMSYS (free) | lmsys.org / lmarena.ai | Auxiliary |
| vLLM (free) | vllm.ai | Practice |
| Experiment tracking with Weights & Biases (free tier) | wandb.ai | Auxiliary |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Treating API access as understanding | Language models are most commonly accessed through APIs, and practitioners can develop sophisticated workflows without understanding what the model is doing. When the model fails in unexpected ways — inconsistent behavior, prompt sensitivity, hallucination on specific topics — API-level understanding provides no diagnostic capability. | Study the architecture, pretraining, and alignment pipeline before building complex applications. Karpathy’s nanoGPT and the Stanford CS336 materials are the right paths to genuine understanding alongside practical API work. |
| Confusing benchmark performance with deployment capability | Benchmark performance on MMLU, HumanEval, or similar is not the same as performance on your specific task with your specific data and user population. Models that rank highly on leaderboards may fail on domain-specific tasks, on unusual prompting patterns, or in the face of adversarial user inputs. | Evaluate models on representative samples of your actual deployment inputs before committing to them. Define the failure modes you care about and test for them specifically. Use LM-Evaluation-Harness for standardized comparison, but treat it as a starting point rather than a final answer. |
| Treating hallucination as a fixable bug | Hallucination — plausible-sounding but incorrect output — is structural to how language models work: they are trained to produce fluent text that is statistically consistent with their training data, not to verify factual accuracy. Techniques like RAG, fine-tuning on accurate data, and self-consistency reduce hallucination frequency but do not eliminate it. | Design systems to handle hallucination architecturally: add retrieval for knowledge-grounded tasks, add verification steps where correctness matters, communicate uncertainty to users, and do not deploy in settings where undetected errors are catastrophic. Treat hallucination reduction as risk management, not defect elimination. |
| Underestimating the importance of pretraining data | Model behavior is substantially determined by pretraining data. A model’s knowledge cutoff, its factual biases, its performance on low-resource languages, and its behavior on domain-specific content are all functions of what was in the training data. Practitioners who treat the model as a fixed capability without understanding its training distribution make predictable errors: assuming capabilities the model does not have because the domain was underrepresented, or assuming properties that were characteristics of the training distribution rather than generalizable capabilities. | Develop familiarity with what major models were trained on and what their training data implies. For domain-specific applications, characterize what fraction of training data was relevant to your domain before inferring capability claims from general benchmarks. |
| Dismissing alignment as a separate concern | Alignment — shaping model behavior through RLHF, DPO, or similar methods — is not separate from performance; it is constitutive of it. Models without alignment behave as next-token predictors trained on internet text, which is very different from what helpful, deployed models do. The difference between a base model and an instruction-tuned aligned model is enormous in practice, not in the architecture or weights alone. | Understand the alignment pipeline — SFT, reward modeling, preference fine-tuning — as engineering that shapes the model’s behavior space, not as an add-on. Study the failure modes of alignment (sycophancy, reward hacking, inconsistency) as failure modes of the deployment system, requiring the same engineering attention as capability failures. |
| Treating open-weight models as transparent | Open-weight models provide access to weights and sometimes to training details, but they are not fully transparent: training data is often undisclosed or partially disclosed, fine-tuning procedures are often proprietary, and the relationship between weights and behavior is opaque without interpretability tools. Practitioners who treat “open weights” as equivalent to “transparent” make unjustified inferences about model provenance and behavior. | Distinguish clearly between access levels: API-only access, open weights with partial training information, and genuinely open models with full training disclosure. Interpretability research (§5.6) addresses the transparency question at the mechanistic level; use it as appropriate for applications where understanding model behavior is critical. |
中文
语言模型是定义在 token 序列上的概率分布。给定一串前面的 token，它会为每一个可能的下一个 token 分配概率。这就是任务的完整规格：预测下一个 token。大语言模型所做的其他一切——回答问题、编写代码、推理问题、在语言之间翻译、总结文档——都来自于在极其多样的文本上训练模型，使它能够很好地预测 token。一个能够很好预测互联网文本中下一个 token 的模型，已经学到了某些关于产生这些文本的世界的东西。
基础模型是大型预训练模型，其学到的表示可以迁移到广泛的下游任务，而不需要针对每个任务从零开始训练；大语言模型是其中最显眼、也最具工业重要性的实例，但这个类别还包括多模态模型、视觉模型、语音模型、生物模型和控制模型。“基础”（foundation）这个词指向的正是这种通用性：一个预训练模型可以通过微调、提示，或二者结合，成为许多应用的基础。这是对前一个深度学习十年的结构性改变；在那时，每个任务通常都需要一个为该任务专门训练的模型。
大语言模型之所以应当从 §5.3（深度学习）中单独分出一节，是因为规模，以及规模带来的行为质变已经在实践中变得可用。深度学习提供了架构底座——Transformer——以及训练方法。LLM 增加的是数据量（数万亿 token）、模型规模（数十亿到数万亿参数），以及在规模上变得可靠得多的能力：上下文学习、思维链提示、代码生成、指令遵循和多步工具使用。这些行为并不是作为独立模块被工程化出来的；它们来自架构、规模、训练数据和后训练之间的相互作用。
前置知识：深度学习（§5.3）——Transformer、自注意力和大规模训练。机器学习基础（§5.2）——评估方法论，尤其是基准表现与真实能力之间的区别。强化学习（§5.4）——RLHF 会贯穿使用 RL 概念。
从词向量到通用系统
通向大语言模型的道路，经过了一系列表示形式：从基于计数的统计，到学到的词向量，到上下文嵌入，再到 Transformer。每一步都提升了语言表示的表达能力。
2013 年以前的古典 NLP 使用的是稀疏的、人工设计的特征表示——词袋、n-gram 计数、句法树。其关键局限在于，这些表示把词当成彼此独立的符号：“bank” 在金融语境中和 “bank” 在地理语境中是同一个特征。对于短文本分类任务，这种方式足够；但对于需要跨语境理解意义的任务，它就不够了。
Mikolov 等人的 word2vec（2013）表明，一个简单的预测任务——从邻近词预测某个词，或从某个词预测邻近词——在大规模语料上训练后，会产生具有惊人几何性质的词嵌入：“king” 的向量减去 “man” 再加上 “woman”，会接近 “queen” 的向量。词语的语义内容被编码进了嵌入空间的几何结构中。word2vec 的工作方式是压缩：一个神经网络如果被迫用一个低维向量来预测上下文，就必须高效编码词语的分布语义。同一年，Sutskever、Vinyals 和 Le 展示了用于机器翻译的序列到序列模型，证明编码器—解码器网络可以把变长输入序列映射成变长输出序列。
Bahdanau、Cho 和 Bengio 于 2015 年引入的注意力机制，解决了序列到序列模型中的瓶颈：把整个源序列压缩成一个固定大小的向量，会在长序列中丢失信息。注意力允许解码器在每一个解码步骤中，有选择地关注编码器输出的不同部分，从而显著改善长句子的机器翻译效果。
Transformer 架构（Vaswani 等，“Attention Is All You Need”，2017）彻底去除了循环结构，用多头自注意力取而代之，使每个位置都可以同时关注其他所有位置。相比循环架构，Transformer 有两个主要优势：它可以并行训练（没有序列依赖），因此在现代硬件上训练效率高得多；并且它的全对全注意力能够捕捉循环架构难以处理的长距离依赖。Transformer 最初被提出为一种更好的机器翻译模型。一年之内，它重塑了整个 NLP 领域，并很快影响了深度学习的其他部分。
BERT（Devlin 等，2018）展示了使用双向 Transformer 进行大规模预训练的力量。它不是像自回归模型那样预测下一个词（从左到右预测），而是使用掩码语言建模：随机遮蔽 15% 的 token，并训练模型根据周围上下文预测这些被遮蔽的 token。先在大规模语料上预训练，再在下游任务上微调，会在所有主要 NLP 基准上显著超过此前架构。BERT 确立了“先预训练，再微调”的范式：先在无标签数据上预训练大模型，然后以较低成本把它适配到具体任务。
GPT（Radford 等，2018）探索了另一个方向：自回归预训练，即根据前面所有 token 预测下一个 token，并且使用了此前未曾尝试过的更大规模。GPT-2（2019）把这一方法扩展到 15 亿参数，表明训练用来预测文本的语言模型，能够生成出人意料地连贯的长文本内容，并能零样本迁移到下游任务。OpenAI 曾短暂拒绝发布 GPT-2 的权重，理由是担心被滥用；这个决定既招致批评，也吸引了注意，并预示了随着能力增长而不断加剧的争论。
GPT-3（Brown 等，2020）进一步扩展到 1750 亿参数，并揭示出一个此前没有被预料到的东西：上下文学习。给定提示中的少量任务示例——“Translate English to French: Sea otter → loutre de mer. Question → ”——GPT-3 可以在不进行任何参数更新的情况下完成翻译，只是通过匹配上下文中的示例模式来执行任务。这不是被编程进去的；它从规模中涌现出来。原本只是在下一个 token 预测上表现足够好的模型，变成了看起来更像通用问题求解器的东西。GPT-3 论文的影响立刻显现：它掀起了一波至今未消退的产业投资、研究关注和公众对语言模型的认知。
Kaplan 等人的 “Scaling Laws for Neural Language Models”（2020）为后续发展提供了组织原则。语言模型表现会随着计算量、模型规模和数据规模按平滑幂律提升，并且没有可见的饱和迹象。如果这个幂律成立，那么对规模的投资就会继续带来成比例的回报。Hoffmann 等人的 “Training Compute-Optimal Large Language Models”（2022，即 Chinchilla 论文）对此进行了修正：Kaplan 定律相对于计算量系统性地低估了数据量，导致模型参数过多、数据训练不足。Chinchilla 拥有 700 亿参数和 1.4 万亿 token，而 Gopher 拥有 2800 亿参数和 3000 亿 token；Chinchilla 以显著更低的推理成本匹配或超过了 Gopher 的性能。这篇论文改变了这个领域分配计算预算的方式，也重新确立了数据质量和数据数量与模型规模同样重要。
对齐转向出现在 InstructGPT（Ouyang 等，2022）中。一个被训练来预测互联网文本的 GPT-3 规模模型，虽然有能力，但并不可靠：它会以看似合理的方式续写提示，却未必在具体语境中有帮助，甚至可能有害。RLHF 微调通过以下方式处理这一点：先根据人类对模型输出的偏好训练一个奖励模型，然后使用 PPO 微调语言模型，使其最大化该奖励。尽管 InstructGPT 小于基础版 GPT-3，却在人类评估者眼中明显更有帮助。这一技术——大规模预训练、监督微调、根据人类偏好训练奖励模型、RL 微调——成为部署语言模型的标准流水线。ChatGPT 使用了它；GPT-4 使用了它；Claude 使用了它；几乎每一个已部署的前沿模型都使用它或非常接近的变体。
2022–2023 年竞争性生态的出现，带来了几项同时发生的发展。PaLM（Chowdhery 等，2022）证明，在高质量数据上扩展到 5400 亿参数，可以让模型在若干推理基准上突破此前性能上限。LLaMA（Touvron 等，Meta，2023）表明，在更多数据上训练的小模型，在推理时可以与更大的模型竞争——这相当于把 Chinchilla 缩放思想应用到开放权重发布中。LLaMA 的开放权重催化了一个研究者共同体，使他们能够微调、探测和修改接近前沿规模的模型，并在数月内产生了几十个衍生模型。
直接偏好优化（DPO，Rafailov 等，2023）简化了 RLHF 流水线。它表明，RL 步骤可以被一个监督目标取代，该目标直接把奖励参数化为策略概率与参考概率之间的对数比。DPO 用更简单的训练流程产生了与 RLHF 相当的结果，而且不需要单独的奖励模型或 RL 基础设施。
推理模型时代始于 OpenAI 在 2024 年发布 o1，随后是 o3、DeepSeek-R1，以及 Anthropic 和 Google 的类似模型。这些模型会在给出最终回答之前，为隐藏的审慎轨迹或多步生成分配更多测试时计算——这是一种测试时计算缩放，用推理时间换取困难推理任务上的准确率。它们在数学和代码基准上的结果很惊人：此前最先进模型难以解决的问题，在使用扩展推理的模型面前变得更可处理。其机制——这到底是否构成某种有意义的“推理”，还是只是复杂的模式补全——仍在激烈争论中。
围绕语言模型建立的基础设施已经成熟为一个生态系统。Hugging Face 平台通过统一 API 使数百个开放权重模型可用。vLLM 和 TensorRT-LLM 提供优化推理。参数高效微调方法（LoRA、QLoRA）允许在消费级硬件上微调大模型。检索增强生成（RAG）系统把模型输出锚定在检索到的文档中。智能体框架编排多步交互。2017 年作为一个研究方向开始的东西，现在已经成为一个产业的基础设施。
架构、训练、适配与评估
Transformer 及其缩放性质
Transformer 的核心操作是多头自注意力。每个 token 表示会被投影成 query、key 和 value；注意力权重通过 query 和 key 之间缩放点积的 softmax 计算；value 则用这些权重聚合。多头注意力会在多个子空间中独立运行这一过程，并把结果拼接起来。一个 Transformer 层会把自注意力、逐位置前馈网络和层归一化堆叠在一起。
自回归语言模型——GPT 风格——使用带因果掩码的自注意力，确保每个位置只能关注它之前的位置。它们使用下一个 token 预测的交叉熵损失训练；如 §3.8 所讨论的，这等价于最小化模型分布与数据分布之间的 KL 散度。这意味着模型被训练成去近似训练语料中的文本分布，包括其中所有统计规律、事实内容和风格模式。
缩放性质并不是任意的。随着模型参数增加，它们在保留文本上的表现会按幂律提升，每增加一个数量级参数，指数约为 0.07（Kaplan 等）。但有效行为会在某些规模上发生质变：少样本学习（GPT-3 水平）、可靠的指令遵循（InstructGPT 水平）、复杂多步推理（推理模型水平）。这些转变并不平滑——在特定能力评估中，它们近似表现为阶跃函数。这造成了一个根本预测难题：在小规模上不存在的能力，可能会在更大规模上突然出现；而在大规模上可见的能力，也未必能从小规模实验中预测出来。
注意力机制在序列长度上的二次计算和内存成本，是上下文窗口长度的主要约束。Flash Attention（Dao 等，2022）及其后续方法通过分块处理注意力，使精确注意力的内存使用与序列长度成正比，而不是与其平方成正比。上下文窗口从 GPT-3 的 2K token 扩展到当前模型的数百万 token，依赖于架构修改（旋转位置嵌入、具有次二次复杂度的注意力变体）和工程改进。
预训练、微调与对齐
预训练需要极大规模的数据。开放权重模型使用的 C4、Common Crawl 和 Pile 数据集包含数千亿到数万亿 token。在这个规模上，数据质量非常重要：去重（近重复文档主导了网络，并降低数据效率）、质量过滤（识别高质量文本的分类器）和数据组成（领域混合——网页、代码、书籍、科学论文——会影响模型学到什么）都是活跃的工程决策。使用代码训练的模型，比只在文本上训练的模型表现出更好的形式推理能力，这也是所有前沿模型都会训练大量代码的原因。
指令微调会把一个预训练基础模型放到（指令，回答）对上训练——这些示范展示模型如何有帮助地回答问题并遵循指令。这会把模型的默认行为从“继续文本”改变为“回应指令”。指令遵循示范的质量很重要：SELF-INSTRUCT（Wang 等，2023）表明，模型可以通过提示更大模型来生成自己的指令数据，从而在不进行大量人工标注的情况下扩展指令微调数据。
RLHF 加入了来自人类比较的偏好信号。给定同一个提示下的两个模型回答，人类标注者指出哪一个更好。根据这些比较训练出的奖励模型，会为任意回答学习一个标量奖励。然后语言模型使用 PPO 进行微调，以最大化这个奖励，同时通过 KL 惩罚保持接近参考策略。KL 惩罚可以防止奖励黑客：如果没有它，语言模型会找到满足奖励模型标准、但偏离合理语言的回答。奖励最大化与 KL 惩罚之间的权衡，是 RLHF 的关键超参数。
RLHF 的已知失败模式已经被充分记录：迎合性（模型为了最大化认可而同意用户，而不是追求准确）、冗长偏置（奖励模型偏好更长回答）和过度优化（Goodhart 定律——当奖励模型被不断优化时，模型会找到满足奖励模型、却违背原本奖励精神的回答）。DPO 的关键优势，是它不需要显式奖励模型，也不使用 RL，因此可以部分避免这些失败模式。
评估：模型能做什么，不能做什么
评估语言模型能力确实很难。模型会在某些基准上表现很强，却在看似相似的任务上失败。性能对提示框架很敏感。基准污染——训练数据中包含测试项——会抬高基准分数。而从基准表现映射到部署能力，也并不可靠。
标准基准测量不同能力维度：MMLU 测试 57 个学科的知识；HumanEval 和 MBPP 测试代码生成；GSM8K 和 MATH 测试数学推理；BBH（Big-Bench Hard）测试具有挑战性的推理任务；MT-Bench 评估指令遵循。每个基准都有已知失败模式：MMLU 可以通过表面模式匹配解决，HumanEval 测的是相对简单的函数，GSM8K 已基本被当前前沿模型解决。
校准——模型置信度是否与准确率相关——是一个不同于准确率的评估维度。一个错误时会说“我不确定”、正确时会说“我有把握”的模型，比一个在正确和错误时都同样自信的模型校准得更好。当前模型用自然语言表达的不确定性并没有很好校准；校准过的 token 概率更可靠，但在已部署系统中更难访问。
评估已部署系统的行为，需要红队测试：系统性尝试诱导模型产生有害、错误或违反政策的输出。红队测试已经形成了一套关于失败模式的文献——提示注入攻击、越狱、不一致的价值对齐、迎合性——它补充了基准评估。安全评估包括对危险能力的测试，例如生物武器合成、网络攻击协助等，这些必须在部署前检查。
学习这一部分会改变什么
大语言模型会改变实践者对计算能做什么，以及不能被信任去做什么的理解。
第一个变化，是理解能力—可靠性差距。当前大语言模型可以做很多事情，但无法在高风险场景中可靠保证正确性。学习过其架构、训练和失败模式的实践者会理解原因：模型近似的是训练数据中的文本分布，这意味着它会生成听起来可信、但可能在事实层面错误的输出。这会塑造使用语言模型的系统应如何设计——在正确性重要的地方，需要验证、锚定、人类监督和优雅降级。
第二个变化，是评估纪律。基准表现不是部署表现。学习过评估文献的实践者知道，应当在具体部署输入上测试模型，应当刻画失败模式，而不是只看平均表现；应当同时评估校准和准确率，并对看起来好得过分的结果保持怀疑。这种纪律可以避免一种常见失败：仅仅因为模型在无法迁移的基准上表现好，就把它部署出去。
第三个变化，是架构透明性。理解自注意力、位置编码的作用、前馈层的功能，以及上下文窗口限制如何产生的实践者，能够从架构约束角度解释模型行为。为什么模型在很长上下文上表现下降？为什么模型无法精确计数？为什么某些提示模式能改善表现？这些问题有架构层面的答案；学习过架构的实践者能够真正处理这些问题。
第四个变化，是能够做出校准过的部署决策。哪些任务值得微调，而不是只用提示？什么时候 RAG 比更长上下文更合适？什么时候 7B 开放权重模型够用，什么时候需要前沿模型？哪些应用需要哪些安全缓解措施？这些决策需要理解能力、成本、可靠性、隐私和控制之间的权衡；这种理解来自学习，而不是来自调用 API。
资源
书籍与文本
基础性的 Transformer 文本现在主要是论文，而不是书籍——这个领域的发展速度超过了书籍出版周期。不过，Jurafsky 和 Martin 的 Speech and Language Processing（第 3 版，可在 web.stanford.edu/~jurafsky/slp3 免费在线获取）已经更新，覆盖了神经语言模型、Transformer 和大语言模型，并且仍然是最完整的教材式处理。它对 Transformer 时代之前古典 NLP 的覆盖，也提供了有价值的背景。
对于机器学习和深度学习的基础准备，Bishop 和 Bishop 的 Deep Learning: Foundations and Concepts（2024，§5.3 参考）以及 Goodfellow 等人的 Deep Learning（免费，§5.3 参考）提供了必要背景。
Bommasani 等人的 “On the Opportunities and Risks of Foundation Models”（Stanford CRFM，2021，免费）是一份长报告，而不是一本书，但它是目前最全面阐述基础模型是什么、能做什么、会带来什么风险的单一文档。具体内容已有些过时，但它仍然提供了最系统的框架。
一手文献是权威来源。最重要的论文包括：Vaswani 等 2017（Transformer）、Devlin 等 2018（BERT）、Brown 等 2020（GPT-3）、Kaplan 等 2020（缩放定律）、Hoffmann 等 2022（Chinchilla）、Ouyang 等 2022（InstructGPT）、Rafailov 等 2023（DPO）。每篇大约 10–30 页，具备 §5.3 前置知识的读者可以直接阅读。
| 资源 | 作用 | 类型 |
|---|---|---|
| Jurafsky & Martin, Speech and Language Processing（第 3 版，免费） | 语言模型和 Transformer 基础 | 入门 |
| Bommasani et al., “On the Opportunities and Risks of Foundation Models”（免费） | 综合框架报告 | 入门 |
| Vaswani et al., “Attention Is All You Need”（2017，免费） | Transformer 里程碑论文 | 深入 |
| Brown et al., GPT-3 paper（2020，免费） | 少样本学习；开启时代 | 深入 |
| Kaplan et al., scaling laws paper（2020，免费） | 缩放关系 | 深入 |
| Hoffmann et al., Chinchilla paper（2022，免费） | 计算最优训练 | 深入 |
| Ouyang et al., InstructGPT paper（2022，免费） | RLHF 对齐流水线 | 深入 |
| Rafailov et al., DPO paper（2023，免费） | 直接偏好优化 | 深入 |
| Transformer architecture reference: Phuong & Hutter（2022，免费） | 精确架构参考 | 参考 |
| Chip Huyen, AI Engineering（付费） | 基础模型应用工程 | 实践 |
课程与讲座
Stanford CS336（Language Modeling from Scratch，材料见 cs336.stanford.edu）是目前可获得的、关于训练语言模型的技术上最严谨课程，覆盖数据工程、架构、训练基础设施和评估。它面向的是想理解前沿模型如何被构建，而不只是如何使用模型的实践者。
Andrej Karpathy 的 Neural Networks: Zero to Hero（§5.3 参考）最后会训练一个字符级 GPT，从第一性原理建立对自回归语言模型的直觉。
Hugging Face NLP Course（huggingface.co/course 免费）提供实践基础：使用 tokenizer、预训练模型、微调和 Transformers 库。对于构建应用的实践者，这是合适的实践入口。
fast.ai 的 Practical Deep Learning 及其配套课程包含关于语言模型的课程，强调实践实现。
| 课程 | 平台 | 类型 |
|---|---|---|
| Stanford CS336 Language Modeling from Scratch | cs336.stanford.edu | 深入 |
| Berkeley LLM Agents course（免费） | rdi.berkeley.edu | 深入 |
| Hugging Face NLP Course（免费） | huggingface.co/course | 实践 |
| Hugging Face Agents Course（免费） | huggingface.co/learn | 实践 |
| Cohere LLM University（免费） | docs.cohere.com | 辅助 |
| Karpathy LLM-adjacent modules from Neural Networks: Zero to Hero（免费） | YouTube | 辅助 |
实践、工具与当前资料
Karpathy 的 nanoGPT（GitHub，免费）用大约 300 行 PyTorch 实现了完整的 GPT 训练流水线。在 Shakespeare 规模或 Wikipedia 规模文本上训练它，然后修改架构或训练过程，是在可达规模上参与语言模型训练最有教学价值的实践方式。
Hugging Face 生态系统——Transformers、Datasets、PEFT、TRL——是使用语言模型的标准基础设施。熟悉 Transformers API（加载模型、运行推理、使用 PEFT/LoRA 微调）是大多数应用工作的实践前提。TRL 库提供了 RLHF 和 DPO 训练实现。
LM-Evaluation-Harness（EleutherAI，GitHub 免费）提供了跨数十个基准的语言模型标准化评估。用它评估一个模型并检查各任务表现，可以揭示总分所掩盖的能力异质性。
vLLM（免费）通过 PagedAttention 提供高吞吐量的大语言模型推理，使高效服务更大模型成为可能。理解 PagedAttention 如何管理 KV cache，有助于理解推理工程。
OpenAI API、Anthropic API、Google AI API 提供对闭源权重前沿模型的访问。系统比较不同提供商的模型行为——针对具体任务类型、失败模式和提示策略——可以建立 API 文档无法提供的校准理解。
| 资源 | 平台 | 类型 |
|---|---|---|
| Karpathy, nanoGPT（免费） | GitHub | 实践 |
| OpenAI Cookbook（免费；API 使用可能付费） | GitHub / cookbook.openai.com | 实践 |
| Anthropic Cookbook（免费；API 使用可能付费） | GitHub | 实践 |
| Prompt Engineering Guide（免费） | dair.ai / promptingguide.ai | 辅助 |
| Hugging Face Transformers / TRL（免费） | huggingface.co | 实践 |
| LM-Evaluation-Harness（免费） | GitHub / EleutherAI | 实践 |
| HELM（免费） | crfm.stanford.edu/helm | 参考 |
| Chatbot Arena / LMSYS（免费） | lmsys.org / lmarena.ai | 辅助 |
| vLLM（免费） | vllm.ai | 实践 |
| Experiment tracking with Weights & Biases（免费层） | wandb.ai | 辅助 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 把 API 使用当成理解 | 语言模型最常通过 API 访问，实践者可以在不理解模型做了什么的情况下，发展出复杂工作流。当模型以意外方式失败时——行为不一致、对提示敏感、在特定主题上幻觉——API 层面的理解无法提供诊断能力。 | 在构建复杂应用之前，学习架构、预训练和对齐流水线。Karpathy 的 nanoGPT 和 Stanford CS336 材料，是在实践 API 工作之外获得真正理解的正确路径。 |
| 把基准表现混同于部署能力 | MMLU、HumanEval 或类似基准上的表现，不等于模型在你的具体任务、具体数据和具体用户群体上的表现。排行榜上排名很高的模型，可能在领域特定任务、不常见提示模式或对抗性用户输入面前失败。 | 在确定使用模型之前，用真实部署输入的代表性样本进行评估。定义你关心的失败模式，并专门测试它们。用 LM-Evaluation-Harness 做标准化比较，但把它当成起点，而不是最终答案。 |
| 把幻觉当成可以修掉的 bug | 幻觉——听起来可信但不正确的输出——是语言模型工作方式中的结构性问题：它们被训练来生成与训练数据在统计上相容的流畅文本，而不是验证事实准确性。RAG、在准确数据上微调、自一致性等技术可以降低幻觉频率，但不能消除它。 | 从架构上设计系统来处理幻觉：在需要知识锚定的任务中加入检索，在正确性重要的地方加入验证步骤，向用户传达不确定性，不要部署到未被发现的错误会造成灾难性后果的场景。把幻觉降低视为风险管理，而不是缺陷消除。 |
| 低估预训练数据的重要性 | 模型行为在很大程度上由预训练数据决定。模型的知识截止时间、事实偏差、低资源语言表现，以及在特定领域内容上的行为，都取决于训练数据中有什么。实践者如果把模型当成固定能力，而不理解其训练分布，就会犯可预期的错误：因为某个领域在训练中代表不足，却假设模型具备该能力；或者把训练分布的特征误认为可泛化能力。 | 熟悉主要模型训练过什么，以及训练数据意味着什么。对于领域特定应用，在根据通用基准推断能力之前，先刻画训练数据中有多少比例与该领域相关。 |
| 把对齐看成独立问题 | 对齐——通过 RLHF、DPO 或类似方法塑造模型行为——并不与性能分离；它构成了性能本身。没有对齐的模型表现为在互联网文本上训练的下一个 token 预测器，这与有帮助的、已部署模型所做的事情非常不同。基础模型与经过指令微调和对齐的模型之间，在实践中差异巨大，而不仅是架构或权重上的差异。 | 把对齐流水线——SFT、奖励建模、偏好微调——理解为塑造模型行为空间的工程，而不是附加项。学习对齐失败模式（迎合性、奖励黑客、不一致性），并把它们视为部署系统的失败模式，用与能力失败同样的工程注意力处理。 |
| 把开放权重模型当成透明模型 | 开放权重模型提供了对权重的访问，有时也提供部分训练细节，但它们并不完全透明：训练数据常常没有公开或只部分公开，微调流程常常是专有的，而没有可解释性工具时，权重与行为之间的关系仍然不透明。把“开放权重”当成“透明”，会让实践者对模型来源和行为做出没有根据的推断。 | 清楚区分访问层级：仅 API 访问、开放权重但训练信息不完整、以及真正开放且完整披露训练过程的模型。可解释性研究（§5.6）在机制层面处理透明性问题；对于理解模型行为至关重要的应用，应当适当使用这些方法。 |