English
A distributed system coordinates work across independent computers by exchanging messages. Partial failure is the starting problem: one component can fail while others continue. Suppose a payment service performs a charge but its reply is lost. The caller’s timeout cannot distinguish that outcome from a request that never arrived. Retrying requires a way to recognize the same business operation, not just a more reliable connection.
Distribution can increase capacity, place data near users, and keep a service running through some failures. Each benefit requires a design: partitioning spreads work, replication preserves copies, and protocols coordinate decisions. Adding machines alone establishes none of these properties. Communication delay, failure handling, and shared dependencies may limit the benefit.
Study three levels together: the system model states timing and failure assumptions; algorithms establish properties such as agreement or convergence under those assumptions; implementations must preserve them across storage, recovery, membership changes, and application behavior. Physical clocks can be useful with explicit uncertainty bounds, but machines do not share a perfectly synchronized clock.
Prerequisites: Operating systems (§4.2), computer networks (§4.3), and databases (§4.4), especially concurrency, message delivery, transactions, and durability.
From Logical Clocks to Globally Distributed Transactions
Distributed-systems theory developed alongside practical needs for communication, shared services, and fault tolerance. The recurring question was how to coordinate without assuming that every component was currently reachable or behaving correctly.
Lamport’s 1978 work separated causal ordering from physical timestamps. If an event precedes another within a process, or a message is sent before it is received, these relations contribute to happens-before. Lamport clocks assign increasing numbers along this relation, but a smaller timestamp does not prove causality. Vector clocks can also distinguish concurrent events. Concurrent events may still update the same business object and conflict; independence in the causal order is not independence of their effects.
The Byzantine generals work of Lamport, Shostak, and Pease examined agreement when faulty participants can send contradictory information. Its thresholds depend on the communication and authentication assumptions. The familiar 3f+1 replica requirement appears in important Byzantine fault-tolerant settings, but is not an impossibility bound for every conceivable protocol and model. Crash tolerance and Byzantine tolerance address different failures.
Fischer, Lynch, and Paterson’s 1985 result concerns deterministic consensus in a fully asynchronous model with even one possible crash. No such protocol guarantees termination in every admissible execution. An indefinitely delayed message is difficult to distinguish from a failed sender. The result does not require protocols to make incorrect decisions: practical designs can preserve safety while requiring additional timing assumptions for progress. Randomized protocols change another premise of the deterministic result.
Paxos, developed by Lamport and published in the 1998 paper “The Part-Time Parliament,” established a method for preserving agreement across competing proposals and failures. Paxos Made Simple explains numbered proposals, intersecting majorities, and the rules for retaining previously accepted values. The retained information matters as much as counting votes: a new proposal cannot freely overwrite an already chosen value.
Google’s GFS (2003) and Bigtable (2006) papers connected distributed storage to concrete workloads: large sequential operations, replicated data, sorted storage, and recovery on clusters. Bigtable helped popularize storage techniques related to the earlier LSM-tree design; it did not originate the LSM-tree. Spanner (2012) combined replication, transactions, and bounded clock uncertainty to provide external consistency, or strict serializability, across transactions. Its clock API supported ordering and commit waiting; it did not eliminate network coordination.
Amazon’s Dynamo (2007) explored a different design for services that prioritized accepting updates during failures. It combined existing ideas, including vector clocks and consistent hashing, with mechanisms such as sloppy quorums and application reconciliation. Concurrent versions could require a business decision about how to merge them. Dynamo should not be confused with the later DynamoDB service or treated as the inventor of vector clocks.
The CAP result, conjectured by Brewer and proved by Gilbert and Lynch, makes a particular limitation precise: during a network partition, a read/write service cannot guarantee both linearizability and a response to every request at a nonfailed node. It is not a general instruction to choose any two of three desirable features. Abadi’s PACELC framework also draws attention to latency and consistency choices during ordinary operation; it is a design framework rather than a replacement proof of CAP.
Raft, presented by Ongaro and Ousterhout in 2014, organized consensus around leader election, log replication, and safety. Its election restrictions help preserve committed entries when leaders change. Cloud platforms subsequently made replicated stores and coordination services widely accessible. Kubernetes was a new system influenced by Google’s experience with Borg, not simply Borg released as open source.
Correctness testing complements protocol proofs. Jepsen-style tests combine concurrent operations with failures and check the resulting history against a specified consistency model. They can expose an implementation that violates its advertised guarantee. Passing sampled executions provides evidence, while a proof establishes a claim within an explicit model; neither substitutes for identifying the application’s actual requirement.
Consensus, Consistency, and the Limits of Distributed Agreement
Separate the questions: which failures are allowed, which observations clients may see, and when the system must make progress. Consensus, replica consistency, and atomic transaction commitment answer related but different questions.
The System Model and What It Determines
A model specifies timing, communication, storage, and failure behavior. In an asynchronous model, there is no known finite bound on process steps or message delay. This is different from saying that every message is permanently lost. A synchronous model supplies bounds; a partially synchronous model can, for example, require usable bounds to hold after an unknown stabilization time. Timeouts are therefore suspicions used to trigger action, not direct observations of another machine’s death.
Crash-stop processes never return. Crash-recovery processes return and must recover protocol state from storage: forgetting an earlier vote can violate a consensus invariant. Byzantine processes can depart arbitrarily from the protocol. A protocol proved safe for crashes does not automatically tolerate forged messages, corrupted storage, or malicious voting.
Within their stated models, Paxos and Raft preserve safety despite arbitrary delays. Progress requires conditions such as a sufficiently stable leader and communication with a working majority. No protocol implementation is “unconditionally safe” outside its assumptions. Explicit assumptions tell you which failures require another protection, such as storage checksums, authentication, or a different agreement protocol.
Consensus: The Central Problem
Consensus asks participants to decide a value while satisfying agreement, validity, and termination under the chosen model. Agreement prevents different decisions; validity constrains which values may be chosen; termination requires an eventual decision. A replicated state machine uses a sequence of agreed commands so replicas can apply the same deterministic transitions. Ordering commands does not automatically deduplicate a client’s retries or make an external payment atomic with the log.
With five replicas, any two majorities of three intersect. That overlap can carry information about an earlier decision into a later round, but overlap alone is insufficient. In Paxos, a proposer obtains promises for a numbered round, learns earlier accepted proposals, and must use the value of the highest-numbered accepted proposal in those replies if one exists. Acceptors retain promises and accepted values across recovery. These rules connect the overlapping votes into a safety argument.
Raft uses a leader and a replicated log. A voter compares a candidate’s last log term and index with its own before granting a vote; a candidate needs a majority. This does not mean the candidate has the longest log of every server. The Raft paper explains why election restrictions and commitment rules together preserve committed entries.
Replication adds network and storage costs. With a stable leader, batching and pipelining can amortize coordination over many commands. Geographic placement changes quorum latency; changing membership requires rules that preserve the needed overlap. These considerations matter both for small coordination stores and high-throughput databases, rather than dividing systems into those that can and cannot use consensus.
Replication and Consistency Models
Replication preserves copies; a consistency model specifies allowed observations. Linearizability makes each operation appear to take effect at one instant between its call and return, while respecting the real-time order of nonoverlapping operations. If a write completes before a read begins, that read cannot return an older overwritten value. Concurrent operations still need some order consistent with the specification.
Synchronous writes to a majority do not alone establish linearizability. Reads must use a protocol that avoids stale results, and an old leader must not continue acting with obsolete authority. Similarly, the inequality R+W>N gives read/write quorum intersection, but version selection, concurrent writes, membership, and recovery determine what that intersection actually guarantees.
Eventual consistency promises convergence when updates stop and the required communication and reconciliation occur. It does not, by definition alone, promise that every replica accepts writes during a partition. Causal consistency preserves causal dependencies in observations; session guarantees such as read-your-writes and monotonic reads describe what a particular client sees. Name the required property rather than using “consistent” as an unexplained label.
State-based CRDTs arrange state and merging so updates can be combined without losing convergence: merge is associative, commutative, and idempotent, and local updates advance state in the specified order. For a grow-only counter, keep one component per replica. States (2,0) and (0,3) merge by componentwise maximum to (2,3), whose total is five. Receiving the same state twice does not count it twice. Operation-based CRDTs instead have operation and delivery requirements; their operations are not universally idempotent.
Convergence does not decide the desired business meaning. A set with concurrent addition and removal needs a defined winner; an inventory counter that eventually merges can still allow overselling during a partition. Enforcing a global bound may require coordination or distributing a limited supply of reservation rights in advance.
Distributed Transactions and the Coordination Cost
Atomic commitment asks whether all participating parts of a transaction commit or all abort. In two-phase commit (2PC), participants first prepare: each records enough durable information to honor a later commit and votes whether it can do so. The coordinator records and communicates the decision. A prepared participant that cannot learn the decision may block, because deciding independently could contradict another participant. Recovery or communication with a participant that knows the decision can resolve the uncertainty.
Consensus and atomic commitment have different jobs. Consensus can replicate a shard’s log or the coordinator’s decision; the transaction still needs agreement among its participating shards about whether their work can commit. The Spanner design combines Paxos replication with two-phase commit. Replicating the coordinator can improve failure recovery, but does not make cross-shard coordination disappear.
Some computations avoid coordination because new information only adds justified results. The CALM principle connects such monotonic computation with coordination-free consistency under its formal assumptions. “An error has been observed” remains true when more observations arrive. “No error occurred in this interval” needs a reason to regard the interval as complete. Stream-processing watermarks (§4.8) address a related practical question about completeness. These distinctions help identify where coordination is required, without treating every distributed application as a transaction over everything.
What Studying This Changes
Start a design review with a property and a failure scenario. For a payment retry, use a durable operation identifier and connect duplicate detection atomically to recording the business result. For a leased lock, consider a paused holder that resumes after expiry: a downstream service can reject an old fencing token only if it actually checks token order. A lock API alone does not protect an unrelated external resource.
Practice on a small replicated service. Record calls and responses, interrupt communication, restart a replica, and check whether the history satisfies the promised model. Measure recovery as well as steady-state throughput. This makes terms such as linearizability, durability, and availability into distinct claims with observable consequences.
Resources
Books and Texts
Designing Data-Intensive Applications organizes distributed data systems around engineering choices. Van Steen and Tanenbaum’s Distributed Systems offers a broader treatment of naming, communication, coordination, replication, and faults. Lynch’s Distributed Algorithms develops explicit models and proofs. Read a topic at the level needed for the claim you are making: implementation decisions and impossibility proofs require different kinds of detail.
| Book | Role | Type |
|---|---|---|
| Kleppmann, Designing Data-Intensive Applications | Contemporary engineering-oriented entry | Entry |
| Tanenbaum & van Steen, Distributed Systems (4th ed., free) | Comprehensive academic reference | Entry |
| Lynch, Distributed Algorithms | Mathematical treatment of distributed algorithms | Depth |
| Cachin, Guerraoui & Rodrigues, Introduction to Reliable and Secure Distributed Programming | Rigorous algorithmic foundations | Depth |
Courses and Lectures
MIT 6.5840 combines paper reading with distributed implementation labs; use a single offering’s instructions and tests together. Kleppmann’s Cambridge lectures provide another route through models, clocks, broadcast, replication, and consistency. Trace a small execution by hand before implementing it, especially across leader change and recovery.
| Course | Platform | Type |
|---|---|---|
| MIT 6.5840 Distributed Systems (free) | MIT / YouTube | Entry |
| Kleppmann Cambridge distributed systems lectures (free) | YouTube | Entry |
Papers and Current Sources
The papers listed below state different claims. Lamport studies event order; FLP proves a limit under a model; Paxos and Raft establish agreement; Dynamo and Spanner describe concrete design choices; Gilbert and Lynch formalize CAP. For each paper, record the assumptions, claimed property, mechanism or proof, and what the evaluation actually measures. Their lengths and prerequisites differ, so there is no single reading-time requirement.
| Paper | Role | Type |
|---|---|---|
| Lamport, “Time, Clocks, and the Ordering of Events” (1978, free) | Logical clocks; foundational paper | Depth |
| Fischer, Lynch, Paterson, FLP impossibility (1985, free) | The central impossibility result | Depth |
| Lamport, “Paxos Made Simple” (2001, free) | Consensus; essential paper | Depth |
| Ongaro & Ousterhout, Raft paper (2014, free) | Understandable consensus | Depth |
| DeCandia et al., Dynamo (2007, free) | Eventual consistency at scale | Depth |
| Corbett et al., Spanner (2012, free) | Global strong consistency | Depth |
| Gilbert & Lynch, CAP proof (2002, free) | The CAP theorem formally | Depth |
Practice, Tools, and Current Sources
Use the Raft lab to connect protocol rules to persistent state and recovery. TLA+ and TLC can specify a model and explore finite instances; model-checking success is limited to the explored configuration, distinct from a general proof. Jepsen analyses and talks show how to record histories, inject faults, and check consistency. Read findings with their version, configuration, and workload, rather than applying an old result to every release.
| Resource | Platform | Type |
|---|---|---|
| MIT 6.5840 Raft lab | pdos.csail.mit.edu/6.5840 | Practice |
| TLA+ with Raft specification | lamport.azurewebsites.net + GitHub | Practice |
| Jepsen analyses (free) | jepsen.io | Reference |
| Jepsen consistency models (free) | jepsen.io/consistency | Reference |
| Gossip Glomers distributed systems challenges (free) | fly.io/dist-sys | Practice |
| Kingsbury distributed systems testing talks (free) | YouTube | Depth |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Assuming remote calls behave like local calls | A timeout leaves uncertainty about whether an operation took effect. | Specify deadlines, retry behavior, and durable duplicate detection where needed. |
| Using CAP as a product classification | The theorem concerns specified properties during partitions, not every feature of a database. | State the operation, consistency model, and partition behavior being compared. |
| Implementing consensus from vote counting alone | Preserved state, election rules, and recovery connect votes across rounds. | Read the protocol specification and check its invariants through failure and restart. |
| Equating replication with atomic cross-shard transactions | Replicas of one shard and participants in a transaction have different decisions to coordinate. | Trace both the replication protocol and the atomic-commit protocol. |
| Assuming convergence preserves every invariant | Convergent replicas can still accept conflicting reservations or stale decisions. | Test the business rule and choose coordination, reservation rights, or suitable conflict semantics. |
| Treating passing tests as a proof | Tests cover selected executions and depend on a correct checker. | Combine failure-injection tests with invariant reasoning and explicit implementation assumptions. |
中文
分布式系统通过交换消息，协调独立计算机上的工作。理解它可以从部分失败开始：一个组件失败时，其他组件仍可能运行。例如，支付服务已经扣款，但回复丢失。调用方发生超时，无法区分“已扣款但没有收到回复”与“请求根本没到达”。重试需要识别同一次业务操作，只有更可靠的连接还不够。
分布式设计可以增加容量、把数据放在用户附近，并在部分故障下继续服务。每种收益都需要相应设计：分区分散工作，复制保留副本，协议协调决策。仅仅增加机器不能建立这些性质；通信延迟、故障处理与共享依赖也会限制收益。
把三个层次结合起来学习：系统模型说明时间与故障假设，算法在这些假设下建立一致决策或收敛等性质，实现则需要在存储、恢复、成员变更与应用行为中保持这些性质。物理时钟在明确的不确定性界限下可以有用，但机器之间没有完全同步的共享时钟。
前置知识：操作系统（§4.2）、计算机网络（§4.3）与数据库（§4.4），尤其是并发、消息传递、事务与持久性。
从逻辑时钟到全球分布式事务
分布式系统理论与通信、共享服务和容错的实际需求共同发展。一个反复出现的问题是：不能假设每个组件当前都可达、都正确运行时，怎样协调工作？
Lamport 在 1978 年的工作区分了因果顺序与物理时间戳。同一进程中事件的先后、消息先发送后接收，共同建立了“发生在先”（happens-before）关系。Lamport 时钟沿这一关系赋予递增编号，但较小的时间戳不能反过来证明因果关系。向量时钟还可以区分并发事件。并发事件仍可能修改同一个业务对象并发生冲突；因果顺序上没有依赖，不代表效果互不影响。
Lamport、Shostak 与 Pease 的拜占庭将军工作，研究故障参与者可能发送矛盾信息时的一致决策。其阈值依赖通信与认证假设。常见的 3f+1 副本要求适用于一些重要的拜占庭容错模型，并非所有协议和模型都无法突破的界限。崩溃容错与拜占庭容错处理的是不同故障。
Fischer、Lynch 与 Paterson 在 1985 年的结果，针对完全异步模型下、哪怕只有一个进程可能崩溃的确定性共识：这类协议无法保证在每个允许的执行中终止。无限延迟的消息难以与发送者失效区分。该结果不要求协议作出错误决策；实际设计可以保持安全性，同时依靠额外的时间假设取得进展。随机化协议则改变了确定性结果的另一个前提。
Lamport 开发、并在 1998 年《The Part-Time Parliament》中发表的 Paxos，给出了在提案竞争和故障中保持一致决策的方法。Paxos Made Simple解释了带编号的提案、相交的多数派，以及保留已接受值的规则。保留什么信息与数票同样重要：新提案不能随意覆盖已经选定的值。
Google 的 GFS（2003）与 Bigtable（2006）论文，把分布式存储连接到具体工作负载：大型顺序操作、数据复制、有序存储，以及集群恢复。Bigtable 推广了与更早的 LSM 树设计相关的存储技术，并非 LSM 树的起源。Spanner（2012）结合复制、事务与有界时钟不确定性，为事务提供外部一致性，也就是严格可串行化。其时钟 API 支持排序与提交等待，并未消除网络协调。
Amazon 的 Dynamo（2007）探索了另一种设计，优先满足服务在故障期间接受更新的需求。它结合了向量时钟、一致性哈希等已有思想，以及宽松仲裁、应用层协调等机制。并发版本可能需要业务规则决定如何合并。Dynamo 不应与后来的 DynamoDB 服务混为一谈，也不是向量时钟的发明者。
Brewer 提出的猜想及 Gilbert、Lynch 的证明，使 CAP 限制得到精确定义：发生网络分区时，读写服务不能同时保证线性一致性，以及对每个非故障节点上的每个请求都作出响应。它并非任意选择三个优点中两个的通用指令。Abadi 的 PACELC 框架还提醒设计者考虑正常运行期间的延迟与一致性选择；它是设计框架，而不是取代 CAP 的另一项证明。
Ongaro 与 Ousterhout 在 2014 年提出的 Raft，把共识组织为领导者选举、日志复制与安全性。其选举限制帮助系统在更换领导者时保留已提交条目。云平台随后使复制存储与协调服务更易获取。Kubernetes 是受 Google 的 Borg 经验影响的新系统，并非直接把 Borg 开源。
正确性测试补充协议证明。Jepsen 式测试把并发操作与故障组合起来，按指定一致性模型检查所得历史，可能发现实现违反所声称的保证。通过抽样执行提供证据，证明则在明确模型内建立结论；两者都不能代替确定应用真正需要什么性质。
Consensus、一致性与分布式一致的限制
分开询问三个问题：允许哪些故障，客户端可以观察到什么，以及系统何时必须取得进展。共识、副本一致性与事务原子提交彼此相关，但回答不同问题。
系统模型及其决定的内容
模型规定时间、通信、存储与故障行为。异步模型没有已知的有限进程步长或消息延迟上界，这与每条消息都永久丢失不同。同步模型给出界限；部分同步模型可以要求在某个未知的稳定时刻之后，有用的界限持续成立。因此，超时是触发行动的怀疑，而不是直接观察到另一台机器已经死亡。
崩溃停止进程不会返回。崩溃恢复进程会返回，需要从存储恢复协议状态：忘记以前的一次投票，就可能破坏共识不变式。拜占庭进程则可能任意偏离协议。只针对崩溃证明安全的协议，不会自动容忍伪造消息、损坏的存储或恶意投票。
在各自规定的模型内，Paxos 与 Raft 在任意延迟下保持安全性。取得进展则需要足够稳定的领导者、与正常多数派通信等条件。协议实现不会在假设之外“无条件安全”。明确假设，才能知道哪些故障需要校验和、认证或另一种一致决策协议等额外保护。
Consensus：中心问题
共识要求参与者在选定模型下作出一个决定，并满足一致性、有效性与终止性。一致性阻止不同决定，有效性限制可选值，终止性要求最终作出决定。复制状态机使用一串已达成一致的命令，使副本能够应用相同的确定性状态转换。排列命令不会自动去重客户端重试，也不会使外部支付与日志形成原子操作。
五个副本中，任意两个各含三个副本的多数派必然相交。交集可以把较早决定的信息带入后续轮次，但只有交集还不够。在 Paxos 中，提议者为一个编号轮次取得承诺，并获知此前已接受的提案；如果回复中存在已接受提案，就必须使用其中编号最高提案的值。接受者还要在恢复后保留承诺与已接受值。这些规则把相交投票连接成安全性论证。
Raft 使用领导者与复制日志。投票者先把候选者最后一条日志的任期和索引与自身比较，再决定是否投票；候选者需要获得多数票。这不表示候选者的日志必须比每台服务器都长。Raft 论文解释了选举限制与提交规则怎样共同保留已提交条目。
复制增加网络与存储成本。领导者稳定时，批处理与流水线可以把协调成本分摊到多条命令。地理位置改变多数派通信延迟，成员变更则需要保持必要交集的规则。这些因素既适用于小型协调存储，也适用于高吞吐数据库，不能简单把系统分成能用与不能用共识的两类。
Replication 与 Consistency Models
复制保留副本，一致性模型规定允许的观察结果。线性一致性使每个操作看起来都在调用与返回之间的某一瞬间生效，并尊重不重叠操作的真实时间先后。如果一次写入完成后读取才开始，该读取不能返回已经被覆盖的旧值。并发操作仍需要有某种符合规范的排列顺序。
同步写入多数派，本身不足以建立线性一致性。读取需要避免陈旧结果的协议，旧领导者也不能继续使用已失效的权限。同样，R+W>N 给出读写仲裁集合的交集，但版本选择、并发写入、成员关系与恢复方式，决定这个交集实际保证什么。
最终一致性承诺：更新停止，并且所需通信与协调发生后，副本最终收敛。仅凭这个定义，不保证每个副本都在分区期间接受写入。因果一致性保持观察中的因果依赖；“读到自己的写入”“单调读”等会话保证，描述特定客户端可以看到什么。应明确所需性质，不能只用“一致”作为未解释的标签。
基于状态的 CRDT 安排状态与合并方式，使更新结合后仍然收敛：合并满足结合律、交换律与幂等性，本地更新沿规定的顺序推进状态。以只增计数器为例，为每个副本保留一个分量。状态 (2,0) 与 (0,3) 按分量取最大值，合并为 (2,3)，总数为五。重复收到同一状态，不会重复计数。基于操作的 CRDT 则具有操作与交付要求，其操作并非普遍幂等。
收敛不会替业务决定所需含义。并发添加与删除的集合需要规定哪种操作胜出；最终合并的库存计数器，仍可能在分区期间超卖。维持全局界限，可能需要协调，或预先分配数量有限的预留权限。
分布式事务与协调成本
原子提交询问：事务的所有参与部分是否一起提交，或者一起中止。两阶段提交（2PC）先让参与者准备：各自持久记录足以履行后续提交的信息，并投票说明能否提交。协调者记录并传达决定。已经准备但无法获知决定的参与者可能阻塞，因为自行决定可能与另一参与者矛盾。恢复，或与已知决定的参与者通信，可以解除这种不确定性。
共识与原子提交承担不同工作。共识可以复制分片日志或协调者的决定；事务仍需要参与分片共同确定其工作能否提交。Spanner 的设计结合了 Paxos 复制与两阶段提交。复制协调者可以改善故障恢复，却不会使跨分片协调消失。
某些计算可以避免协调，因为新信息只会增加已经得到依据的结果。CALM 原则在其形式假设下，把这种单调计算与无需协调的一致性联系起来。“已经观察到错误”不会因为收到更多观察而失效；“这个区间内没有发生错误”则需要理由认为区间已经完整。流处理的水位线（§4.8）处理与完整性相关的实际问题。这些区分有助于找出真正需要协调的位置，而不是把整个分布式应用都视为一个覆盖所有事物的事务。
学习这一部分会改变什么
设计审查可以从一个性质与一个故障场景开始。支付重试需要持久的操作标识，并把重复检测与记录业务结果原子地连接起来。租约锁则需要考虑暂停的持有者在租约到期后恢复：下游服务只有实际检查栅栏令牌的顺序，才能拒绝旧令牌。一个锁 API 本身不能保护无关的外部资源。
可以用小型复制服务练习：记录调用与响应，中断通信，重启副本，再检查历史是否符合承诺的模型。除了稳定运行的吞吐量，也测量恢复过程。这样，线性一致性、持久性与可用性就成为具有可观察后果的不同主张。
资源
书籍与文本
Designing Data-Intensive Applications 围绕工程选择组织分布式数据系统。Van Steen 与 Tanenbaum 的 Distributed Systems 更广泛地讲述命名、通信、协调、复制与故障。Lynch 的 Distributed Algorithms 展开明确模型与证明。应按所要论证的主张选择阅读深度；实现决策与不可能性证明需要不同细节。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Kleppmann, Designing Data-Intensive Applications | 当代工程导向入口 | 入门 |
| Tanenbaum & van Steen, Distributed Systems（第 4 版，免费） | 综合学术参考 | 入门 |
| Lynch, Distributed Algorithms | 分布式算法的数学处理 | 深入 |
| Cachin, Guerraoui & Rodrigues, Introduction to Reliable and Secure Distributed Programming | 严格算法基础 | 深入 |
课程与讲座
MIT 6.5840 把论文阅读与分布式实现实验结合，使用时应让指导与测试来自同一学期。Kleppmann 的 Cambridge 课程 提供另一条学习模型、时钟、广播、复制与一致性的路径。实现前可以手动跟踪一个小型执行，尤其注意领导者更换与恢复。
| 课程 | 平台 | 类型 |
|---|---|---|
| MIT 6.5840 Distributed Systems（免费） | MIT / YouTube | 入门 |
| Kleppmann Cambridge distributed systems lectures（免费） | YouTube | 入门 |
论文与当前资料
下列论文提出不同主张。Lamport 研究事件顺序，FLP 在模型下证明限制，Paxos 与 Raft 建立一致决策，Dynamo 与 Spanner 描述具体设计选择，Gilbert 与 Lynch 形式化 CAP。阅读每篇时，记录假设、所声称性质、机制或证明，以及评估实际测量什么。它们的长度与前置知识不同，没有统一阅读时长要求。
| 论文 | 作用 | 类型 |
|---|---|---|
| Lamport, “Time, Clocks, and the Ordering of Events”（1978，免费） | Logical clocks；奠基论文 | 深入 |
| Fischer, Lynch, Paterson, FLP impossibility（1985，免费） | 中心不可能性结果 | 深入 |
| Lamport, “Paxos Made Simple”（2001，免费） | Consensus；必要论文 | 深入 |
| Ongaro & Ousterhout, Raft paper（2014，免费） | 可理解的 consensus | 深入 |
| DeCandia et al., Dynamo（2007，免费） | 大规模 eventual consistency | 深入 |
| Corbett et al., Spanner（2012，免费） | 全球 strong consistency | 深入 |
| Gilbert & Lynch, CAP proof（2002，免费） | CAP theorem 的形式证明 | 深入 |
实践、工具与当前资料
用 Raft 实验 把协议规则连接到持久状态与恢复。TLA+ 与 TLC 可以规定模型并探索有限实例；模型检查通过只覆盖所探索配置，不同于一般证明。Jepsen 分析与演讲展示如何记录历史、注入故障与检查一致性。阅读发现时要带上版本、配置与工作负载，不能把旧结果套用于每个发布版本。
| 资源 | 平台 | 类型 |
|---|---|---|
| MIT 6.5840 Raft lab | pdos.csail.mit.edu/6.5840 | 实践 |
| TLA+ with Raft specification | lamport.azurewebsites.net + GitHub | 实践 |
| Jepsen analyses（免费） | jepsen.io | 参考 |
| Jepsen consistency models（免费） | jepsen.io/consistency | 参考 |
| Gossip Glomers distributed systems challenges（免费） | fly.io/dist-sys | 实践 |
| Kingsbury distributed systems testing talks（免费） | YouTube | 深入 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 假设远程调用像本地调用 | 超时留下操作是否生效的不确定性。 | 规定截止时间、重试行为，以及需要时的持久去重。 |
| 用 CAP 给产品简单分类 | 定理涉及分区期间的规定性质，并非数据库的每项功能。 | 说明比较的操作、一致性模型与分区行为。 |
| 仅凭数票实现共识 | 保留状态、选举规则与恢复，把不同轮次的投票连接起来。 | 阅读协议规范，在故障与重启中检查其不变式。 |
| 把复制等同于跨分片原子事务 | 同一分片的副本与事务参与者，需要协调不同决定。 | 同时跟踪复制协议与原子提交协议。 |
| 假设收敛保持每项不变式 | 会收敛的副本仍可能接受冲突预留或陈旧决策。 | 检验业务规则，选择协调、预留权限或合适的冲突语义。 |
| 把测试通过当作证明 | 测试覆盖选定执行，并依赖正确的检查器。 | 把故障注入测试、不变式推理与明确实现假设结合。 |