English
Production systems fail. Not occasionally, not because of insufficient care, but continuously and inevitably at scale. Hardware wears out. Software encounters edge cases that testing did not anticipate. Operational changes interact with existing state in unexpected ways. Dependencies behave differently under novel load conditions. The question for any production system is not whether it will fail but how often, how severely, and how quickly it will recover.
Observability is the technical infrastructure and analytical practice for understanding what production systems are doing: the telemetry — metrics, structured logs, distributed traces, continuous profiling — that makes system state visible, and the discipline for using it to diagnose problems. A system is observable if, when it behaves unexpectedly, engineers can determine the cause by examining its external outputs, for any state the system might reach — including states that were not anticipated when the monitoring was designed. This distinguishes observability from monitoring: monitoring asks predetermined questions and alerts when predetermined thresholds are crossed; observability supports investigation of questions that were not known to be relevant until the system demonstrated that they were.
Reliability engineering is the practice of making systems reliably available through deliberate engineering work. It sets explicit, quantified reliability targets, measures achievement against them, systematically learns from failures when they occur, and engineers away the underlying causes of recurring incidents. Its organizational expression, Site Reliability Engineering (SRE), treats production operation as engineering work requiring the same rigor as feature development — not as residual maintenance that developers perform when they are not building new things.
The two subjects belong together. Observability without reliability goals produces data without purpose. Reliability engineering without observability produces goals without the ability to measure progress toward them or diagnose deviations from them. Together they constitute the discipline of keeping production systems working, which is the operational context in which all the other systems subjects in this chapter ultimately matter.
Prerequisites: Operating systems (§4.2), distributed systems (§4.6), and container orchestration (§4.9) provide the substrate being operated. The systems that fail are the systems those sections describe, and reliability engineering is only as good as the practitioner’s understanding of them.
From Uptime Tracking to Engineering Reliability
The idea that software systems should be reliably available is not new — it is as old as production computing. The SAGE air defense network, deployed in the 1950s, was required to operate continuously because its mission demanded it. Early time-sharing systems had uptime requirements because users depended on them. The novelty of the modern discipline is not the goal of reliability but the systematic engineering approach to achieving and measuring it.
The traditional approach to production operation treated reliability as an operational concern separate from software development: operators maintained systems that developers handed over, applying patches, restarting services, and responding to alerts from monitoring systems. The tools were straightforward — ping, uptime monitors, threshold-based alerting. The mental model was mechanical: a service was up or down, monitored by checks that detected the transition. This model worked adequately at small scale and failed as a discipline as systems grew more complex. A service could be up in the mechanical sense (responding to health checks) while serving incorrect results, high latency, or intermittent errors that affected real users. Threshold-based alerting on known failure modes could not detect novel ones. The operational knowledge required to diagnose failures was held by specific humans, was not reproducible or transferable, and evaporated when those humans left.
Google’s formation of the Site Reliability Engineering function is typically dated to 2003, when Ben Treynor Sloss led the first team chartered to apply software engineering principles to production operation. The founding insight was that reliability was an engineering problem, not an operational one: if the systems were the artifacts, the reliability of those systems was a property that could be engineered in, measured quantitatively, improved through deliberate design, and managed with the same rigor applied to correctness or performance. SRE teams at Google were staffed with software engineers, not operations specialists, and were expected to automate away operational toil rather than perform it indefinitely. The practices that emerged — Service Level Indicators, Service Level Objectives, error budgets, blameless postmortems, chaos engineering — were engineering responses to engineering problems, designed to make reliability work explicit, measurable, and improvable.
The discipline went public with the Google SRE book in 2016: Beyer, Jones, Petoff, and Murphy’s Site Reliability Engineering: How Google Runs Production Systems, released free online. The book assembled Google’s SRE practices for the first time in a form the industry could engage with directly, and its influence was immediate and substantial. SRE as a function, SLOs as a contractual mechanism, error budgets as a reliability-velocity tradeoff tool — these practices spread widely over the following years. The Google SRE model was not universally adopted in its original form; most organizations adapted it for their scale and context. But the underlying ideas — explicit quantitative reliability targets, error budgets, systematic postmortem analysis — changed how the industry thought about reliability.
The observability dimension developed separately and faster. Traditional monitoring — threshold-based alerting on metrics like CPU utilization, memory usage, and request rates — was increasingly inadequate for diagnosing failures in distributed systems with many interacting services. The problem was structural: traditional metrics answered questions about aggregate system state but could not answer questions about the experience of individual requests, users, or subpopulations. A service could have a high error rate for a specific user segment while overall error rate metrics looked normal. A specific API endpoint could be slow while average latency looked acceptable.
Charity Majors, who had joined Facebook in 2010 and later founded Honeycomb, articulated what was missing. Her framing — that observability meant being able to understand the internal state of a system from its external outputs, for any state the system might reach — pointed toward high-cardinality, high-dimensionality telemetry that could be sliced and queried arbitrarily, not only against pre-configured dimensions. Structured logs with rich contextual fields, distributed traces that followed individual requests across service boundaries, and event-driven storage that supported arbitrary queries against any combination of fields were the technical substrate this required. The “three pillars of observability” framing — metrics, logs, traces — became the organizing vocabulary of the field, later extended to a fourth pillar with continuous profiling.
Netflix’s Chaos Monkey, released in 2011 as an open-source tool that randomly terminated production instances, formalized a practice that had been implicit in reliability engineering: if you want to know how a system handles failure, induce failure deliberately rather than waiting for it to happen at inconvenient moments. The Principles of Chaos Engineering, published in 2016, codified this into a discipline: define a “steady state” metric, form a hypothesis about how that metric behaves when failures are induced, run controlled experiments, and use the results to find weaknesses before they cause uncontrolled incidents. Chaos engineering moved from a Netflix-specific practice to a broader discipline with its own tooling (Gremlin, Chaos Mesh, AWS Fault Injection Simulator) and community.
Richard Cook’s “How Complex Systems Fail,” written in 1998 for healthcare and aviation safety contexts and freely available online, became one of the field’s foundational texts without ever being specifically about software. Its eighteen principles — that complex systems contain multiple latent failures, that failures require multiple contributing factors to align, that practitioners are simultaneously the initiators of failure and the last line of defense, that the cause of a failure is always the specific combination of conditions that aligned on that occasion — are structural properties of complex production systems that any reliability engineering practice must account for. The blameless postmortem practice, which became a defining cultural element of SRE, is an application of Cook’s insight that proximate human actions are never the cause of complex system failures; the systemic conditions that made those actions possible are.
The most recent consequential development is the standardization of observability instrumentation through OpenTelemetry, a CNCF project that reached production readiness between 2019 and 2021. Before OpenTelemetry, each observability vendor required its own SDK, making instrumentation tightly coupled to vendor choice and vendor switching prohibitively expensive. OpenTelemetry provides vendor-neutral instrumentation that emits to any compatible backend, shifting competition from instrumentation to analysis capability. Combined with continuous profiling becoming sustainable for production use through sampling-based approaches, OpenTelemetry has restructured the observability infrastructure landscape and significantly reduced the cost of establishing and maintaining observability across an organization.
Service Levels, Failure Analysis, and the Practice of Investigation
Reliability Targets: SLIs, SLOs, and Error Budgets
Service Level Indicators (SLIs) measure what a service is actually doing: error rate, latency distribution, availability, throughput. They are the quantitative basis for reliability conversations. The choice of SLI matters: a service can have low error rates but high latency; it can be available but returning incorrect results. The SLI should measure what users actually care about, not what is easy to measure.
Service Level Objectives (SLOs) define what level of SLI performance is acceptable. An SLO of “99.9% of requests succeed within 200ms over a 30-day window” makes the reliability target explicit and measurable. The SLO should be set at the level of reliability that users actually need — not at “100% available” which is neither achievable nor necessary, and not so low that it permits degraded user experience. Setting SLOs requires empirical understanding of user tolerance for specific failure modes.
The error budget is the complement of the SLO: if the SLO requires 99.9% success rate, the error budget is 0.1% — the acceptable failure rate. Error budgets are the mechanism by which reliability engineering makes the reliability-velocity tradeoff explicit and actionable. When the error budget is healthy, the organization can take risks — deploy more aggressively, experiment with new approaches. When it is exhausted, reliability work takes priority over feature development. This structure replaces the implicit negotiation over “are we reliable enough to ship” with a quantitative policy that both development and operations teams have agreed to in advance.
The SLO and error budget framework is often described as simple and is difficult in practice. Setting meaningful SLOs requires knowing what users actually need, which requires measurement and user research that most teams have not done. Enforcing error budgets requires organizational commitment that development teams often resist when it blocks features they want to ship. Measuring SLIs accurately requires observability infrastructure that may not exist. The framework is a reliable engineering tool; making it work in an organization is a reliability engineering and organizational challenge.
The Anatomy of Failures
Cook’s principles capture the structural facts about complex system failures. No failure has a single cause. Every failure is the result of multiple contributing factors — each individually insufficient to produce the failure — aligning on a specific occasion. The proximate cause (the action or component state immediately before the failure) is never the full cause; the conditions that made the system susceptible to that proximate cause are always part of the causal story.
This has immediate implications for postmortem analysis. A postmortem that identifies the proximate trigger — “the engineer deployed a misconfigured rate limiter” — without identifying the contributing conditions — “why was the misconfiguration possible to deploy, why did it not fail validation, why did the monitoring not catch the problem before it affected users, why was the rate limiter configuration not reviewed” — has identified a symptom and stopped. The remediation that addresses the proximate trigger (training the engineer, adding a deployment check) leaves the contributing conditions intact, so similar failures remain possible through different triggers.
Blameless postmortems are technically superior to blame-based ones, not merely kinder. Blame terminates investigation: once a person is identified as responsible, organizational pressure reduces to obtain an apology and move on, before the systemic causes have been identified. Blameless investigation asks of every contributing factor: what made this factor possible? what would prevent this factor from appearing in future incidents through different mechanisms? The answers produce specific, targeted remediations that address the system’s susceptibility rather than the specific trigger.
The five-whys technique and its modern derivatives are tools for driving causal analysis deeper: for each identified cause, ask what caused that cause. Applied systematically, this reliably reaches the systemic conditions that made the failure possible — organizational pressures, design decisions, operational practices — rather than stopping at human actions. The limit of the technique is that it produces linear causal chains in a system where causality is actually a graph; more sophisticated approaches (fishbone diagrams, fault tree analysis) capture non-linear causal structure.
Observability in Practice: The Three Pillars and Beyond
Metrics are aggregated numerical measurements over time: counters (total requests), gauges (current memory usage), histograms (request latency distribution). They are efficient to store and query at high cardinality over time, but they aggregate away per-request detail. Metrics are appropriate for: detecting that something is wrong, measuring trend behavior over time, alerting on threshold violations. They are not appropriate for: understanding why something is wrong, understanding the experience of specific users or requests, investigating novel failure modes that were not anticipated when the metrics were defined.
Structured logs are records of individual events with key-value fields providing context: the user who made the request, the service that processed it, the specific error that occurred, the duration. Structured logs support arbitrary filtering and aggregation after the fact: “show me all errors for user X from service Y in the last hour” is a log query, not a metrics query. They are more expensive to store than metrics but contain the per-request detail that investigation requires. The shift from unstructured logs (lines of text parsed by regex) to structured logs (JSON or similar formats with well-defined fields) was one of the more consequential observability improvements of the past decade.
Distributed traces show the path of a single request through multiple services — the trace is a tree of spans, each span representing work done in one service, with timing, metadata, and relationships between spans. Distributed tracing answers “what services did this request touch, how long did each take, what happened in each” — questions that neither metrics nor logs answer for requests that cross service boundaries. The Dapper paper (Sigelman et al., 2010) described the tracing system Google used internally; Zipkin and Jaeger implemented the same ideas as open-source systems; OpenTelemetry has standardized the instrumentation layer that produces traces across different languages and frameworks.
Continuous profiling adds a fourth dimension: which code was executing during a time window, and at what rate. Sampling-based profiling tools (Pyroscope, Parca) collect stack traces continuously in production with low enough overhead to sustain indefinitely, then aggregate and store them for retrospective analysis. When combined with distributed tracing, continuous profiling provides the full picture: what requests were doing and how the code was executing while they did it. This closes the gap between “service was slow” and “this specific function was the bottleneck.”
What Studying This Changes
Observability and reliability engineering changes how practitioners think about production systems throughout the software development lifecycle, not only during incidents.
The first change is systematic diagnosis under time pressure. Production incidents require forming hypotheses, testing them against telemetry, eliminating alternatives, and converging on a causal explanation while users are affected and organizational pressure to resolve the incident is acute. This capability is learned by doing — reading about incident response does not produce it — but understanding the systematic approach to hypothesis generation and evidence evaluation provides the framework within which experience accumulates productively.
The second change is designing for observability from the beginning. Systems that are difficult to observe during failures are systems that were not designed with observability as a requirement. The code that emits structured logs with rich context, that propagates trace identifiers across service calls, that exposes meaningful health endpoints and cardinality-appropriate metrics — that code reflects design choices made by engineers who knew the system would eventually behave unexpectedly and built in the ability to understand it when it did.
The third change is honest reliability accounting. Setting SLOs that reflect actual user needs rather than what is easy to achieve, measuring against them without softening the metrics, and enforcing error budgets when they are exhausted requires organizational discipline that most teams find uncomfortable. The discomfort is the point: the discipline makes reliability decisions explicit and contestable rather than implicit and unaccountable, which is what allows reliability to be systematically improved.
Resources
Books and Texts
Beyer, Jones, Petoff, and Murphy’s Site Reliability Engineering: How Google Runs Production Systems (O’Reilly, 2016, free at sre.google) is the canonical foundation. It defines SRE as a discipline, covers SLIs, SLOs, error budgets, on-call practices, postmortem culture, and release engineering with the authority of practitioners who built these practices at scale. Some practices are Google-specific — most organizations do not have Google’s staffing ratios or infrastructure — but the underlying principles transfer universally. The companion Site Reliability Workbook (2018, also free at sre.google) provides case studies and worked examples.
Majors, Fong-Jones, and Miranda’s Observability Engineering (O’Reilly, 2022) is the canonical contemporary observability text. Written by practitioners who built Honeycomb and established the modern observability framing, it covers the distinction between monitoring and observability, the technical infrastructure that supports genuine observability (high-cardinality structured events, trace-based investigation), and the analytical discipline for using it. It is the appropriate companion to the SRE books on the observability side.
Forsgren, Humble, and Kim’s Accelerate (IT Revolution, 2018) provides the empirical backing for reliability engineering practices: the DORA research on what engineering practices actually produce reliable, high-velocity software organizations. The four key metrics (deployment frequency, lead time, mean time to restore, change failure rate) and the research connecting them to organizational outcomes make the case for specific reliability practices with evidence rather than advocacy.
Rosenthal, Jones, Evans, and Casey’s Chaos Engineering (O’Reilly, 2020) covers deliberate fault injection as a reliability practice — what chaos engineering is, how to design experiments, what the practice reveals, and how to operate it safely. The Principles of Chaos Engineering (principlesofchaos.org, free) provides the canonical short statement of the discipline.
| Book | Role | Type |
|---|---|---|
| Beyer et al., Site Reliability Engineering (free) | Canonical SRE foundation | Entry |
| Beyer, Murphy et al., The Site Reliability Workbook (free) | Practical SRE companion | Depth |
| Majors, Fong-Jones & Miranda, Observability Engineering | Contemporary observability standard | Entry |
| Forsgren, Humble & Kim, Accelerate | DORA research; empirical reliability practices | Depth |
| Rosenthal et al., Chaos Engineering | Deliberate fault injection | Depth |
| Sridharan, Distributed Systems Observability (free) | Concise observability introduction | Entry |
| Hollnagel, Resilience Engineering in Practice | Academic resilience engineering foundations | Auxiliary |
Courses, Papers, and Current Sources
Cook’s “How Complex Systems Fail” (1998, two pages, free online) is the foundational text for understanding why complex systems fail structurally. It was written for healthcare and aviation safety but is completely applicable to software systems. Reading it before any postmortem course or literature reshapes how the entire subject is understood. Allspaw’s writings on incident response — available through his blog and through the Adaptive Capacity Labs — provide software-specific depth on how engineers actually reason during incidents.
The Dapper paper (Sigelman et al., Google Technical Report 2010, free) is the foundational distributed tracing paper; understanding it illuminates what contemporary tracing systems are doing and why they are structured as they are. The OpenTelemetry documentation (opentelemetry.io, free) is the authoritative reference for vendor-neutral observability instrumentation.
SREcon talks (free on USENIX YouTube) represent the contemporary practitioner state. The USENIX SREcon conference archives include years of talks on reliability practices, incident management, on-call rotation design, and observability tooling by practitioners who operate production systems at scale.
| Resource | Platform | Type |
|---|---|---|
| Cook, “How Complex Systems Fail” (free) | Foundational failure analysis essay | Depth |
| Allspaw’s writings on incident response (free) | adaptivecapacitylabs.com | Depth |
| Sigelman et al., “Dapper” paper (free) | Distributed tracing architecture paper | Depth |
| OpenTelemetry documentation (free) | opentelemetry.io | Reference |
| AWS Builders’ Library (free) | aws.amazon.com/builders-library | Reference |
| Rob Ewaschuk, “My Philosophy on Alerting” (free) | linuxczar.net/sysadmin/philosophy-on-alerting | Auxiliary |
| Principles of Chaos Engineering (free) | principlesofchaos.org | Reference |
| SREcon talks (free) | USENIX YouTube | Reference |
| sre.google (SRE books and supporting material) | sre.google | Reference |
Practice, Tools, and Current Sources
Public postmortems are one of the most concentrated resources in reliability engineering. Cloudflare’s postmortems, GitHub’s, Stripe’s, and Discord’s are unusually technically detailed and well-written. Reading a postmortem and being able to reconstruct the causal chain, identify the contributing conditions, and evaluate the adequacy of the remediation is a reliability engineering skill that develops through practice on real materials. Cloudflare’s blog archives years of postmortems with explanations of BGP misconfigurations, software bugs, and infrastructure failures in productions systems serving hundreds of millions of users.
Prometheus and its ecosystem (Alertmanager, Grafana) are the most widely deployed open-source observability infrastructure. Running Prometheus locally against a test service, defining SLI-based recording rules, and configuring alerts that fire on symptoms rather than causes is the practical entry to metrics-based observability. Grafana’s tutorial documentation provides guided setup.
Jaeger or Zipkin (both free, open-source) provide distributed tracing infrastructure. Instrumenting a multi-service application with OpenTelemetry and collecting traces in Jaeger makes request propagation across service boundaries visible in a way that no amount of reading about tracing can replicate.
The Chaos Monkey and Chaos Mesh open-source tools make controlled failure injection accessible for local and cluster environments. Running controlled chaos experiments — terminating pods randomly, injecting latency, blocking network traffic — and observing the system’s response makes the resilience engineering principles concrete.
| Resource | Platform | Type |
|---|---|---|
| Cloudflare postmortems and incident reports (free) | blog.cloudflare.com | Practice |
| Public postmortems: GitHub, Stripe, Discord (free) | Various | Practice |
| Prometheus + Grafana (free) | prometheus.io / grafana.com | Practice |
| Jaeger / Zipkin (free) | jaegertracing.io / zipkin.io | Practice |
| OpenTelemetry Demo (free) | opentelemetry.io/docs/demo | Practice |
| Chaos Monkey / Chaos Mesh (free) | GitHub / chaos-mesh.org | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| SRE as a checklist of practices | The SRE literature describes specific practices — SLOs, error budgets, postmortem templates, toil reduction — and learners sometimes implement them nominally without the underlying discipline. The result is organizations with SLOs nobody enforces, error budgets never consumed, and postmortems producing no change. The practices work when supported by organizational commitment; they fail as cargo cult. | Study the principles alongside the practices. Understand why error budgets work before implementing them, why blameless postmortems are technically superior rather than merely culturally nicer, why SLO levels should reflect user needs rather than current achievability. The reasoning behind each practice is as important as the practice. |
| Observability as metrics and dashboards | Metrics and dashboards are components of observability infrastructure, not its sum. Teams that treat observability as “we have Grafana dashboards for our services” find themselves unable to investigate novel failure modes that their dashboards were not designed to display. A service can look healthy on all dashboards while specific users experience high error rates or latency, if the dashboards do not show per-user, per-request, or segment-level behavior. | Evaluate observability infrastructure by what questions it can answer, not by how many dashboards it produces. Invest in structured logging with rich context fields and distributed tracing that supports arbitrary queries. Read Majors et al.’s Observability Engineering for the contemporary standard of what genuine observability requires. |
| Alert on everything | The instinct to alert on many conditions, reasoning that more alerts improve detection, produces alert fatigue: too many pages, engineers stop responding effectively, important alerts are lost in noise, on-call rotation becomes intolerable. Engineers who silence alerts are not irresponsible; they are rationally responding to signal-to-noise problems. | Alert only on symptoms that are directly affecting users, that require human action, and that human action can address. Internal conditions whose customer impact is unclear should not produce pages. The SRE book’s chapter on alerting philosophy is the appropriate starting point; the additional framing from the “My Philosophy on Alerting” document by Rob Ewaschuk is also excellent. |
| Postmortems as blame, even gently | Postmortems that identify the engineer who touched the system before the failure — even framed sympathetically as “we need better training” or “we need clearer documentation” — terminate investigation before systemic causes are found. The question is not who was present but what conditions made the system susceptible to the failure through this mechanism. Failure to reach systemic causes means remediations do not address the real problem. | Explicitly train postmortem facilitation around causal systems thinking. For each identified contributing factor, ask: what made this factor possible? What made the system susceptible to it? What would prevent this contributing factor from producing a similar failure through a different mechanism? Cook’s “How Complex Systems Fail” is essential background for this framing. |
| Treating chaos engineering as reckless | Some organizations resist chaos engineering because it involves deliberately causing failures in production systems. The resistance misunderstands the alternative: production systems will fail anyway, and doing so at unpredictable times with unpredictable impacts. Controlled chaos experiments reveal failure modes in known conditions with known blast radius, while improving incident response capability by practicing failure regularly. | Start with chaos engineering in non-production environments to build confidence, then introduce it in production with careful scope limitation (canary populations, business hours, explicit rollback mechanisms). The goal is not to create incidents but to find failure modes before they create uncontrolled incidents. |
| Using MTTF and MTTR as primary metrics without SLO context | Mean Time To Failure and Mean Time To Recovery are useful aggregate measures but obscure whether reliability targets are being met. A service with long MTTF but severe user impact when it fails may have worse reliability from the user perspective than a service with shorter MTTF but fast recovery and graceful degradation. | Define reliability in terms of user-facing SLIs and SLOs. Measure what users experience, not what the system does internally. MTTR is useful as a component of error budget consumption but should be interpreted in the context of the SLO it affects, not as a standalone metric. |
中文
生产系统会失败。不是偶尔失败，也不是因为不够谨慎，而是在规模化之后持续且不可避免地失败。硬件会磨损。软件会遇到测试未预料到的边缘情况。运维变更会以意外方式与既有状态交互。依赖项会在新的负载条件下表现不同。对任何生产系统来说，问题都不是它是否会失败，而是失败多频繁、多严重，以及恢复速度有多快。
可观测性是用于理解生产系统正在做什么的技术基础设施和分析实践：telemetry——metrics、structured logs、distributed traces、continuous profiling——使系统状态可见，而相应纪律则用于使用这些信息诊断问题。当系统出现意外行为时，如果工程师可以通过检查其外部输出确定原因，并且这种能力适用于系统可能到达的任何状态——包括 monitoring 设计时没有预料到的状态——这个系统就是 observable 的。这一区分了 observability 和 monitoring：monitoring 提出预先确定的问题，并在预先确定的阈值被越过时发出警报；observability 则支持调查那些直到系统表现出相关性之前都不知道需要提出的问题。
可靠性工程是通过有意识的工程工作，使系统可靠可用的实践。它设定明确、量化的可靠性目标，测量目标达成情况，在失败发生时系统性学习，并通过工程手段消除 recurring incidents 的底层原因。它的组织形式 Site Reliability Engineering（SRE），把生产运维视为需要与功能开发同等严谨性的工程工作——而不是开发者在不构建新功能时顺手做的残余维护。
这两个主题属于同一处。没有可靠性目标的可观测性，会产生没有目的的数据。没有可观测性的可靠性工程，会产生无法测量进展、也无法诊断偏离的目标。二者合在一起，构成了维持生产系统正常工作的纪律；而这正是本章其他所有系统主题最终发生作用的运维语境。
前置知识：操作系统（§4.2）、分布式系统（§4.6）和容器编排（§4.9）提供了被运维的基底。失败的系统正是这些章节所描述的系统，而可靠性工程的质量，取决于实践者对它们的理解质量。
从 Uptime Tracking 到工程化可靠性
软件系统应当可靠可用，这个想法并不新——它与生产计算一样古老。1950 年代部署的 SAGE air defense network 必须持续运行，因为其任务要求如此。早期 time-sharing systems 有 uptime 要求，因为用户依赖它们。现代纪律的新意不在于可靠性目标本身，而在于实现和测量可靠性的系统性工程方法。
传统生产运维方式把可靠性视为与软件开发分离的运维关切：operators 维护开发者交付的系统，打补丁、重启服务，并响应 monitoring systems 发出的 alerts。工具很直接——ping、uptime monitors、threshold-based alerting。心智模型是机械式的：一个服务要么 up，要么 down，由检测这种转变的 checks 监控。这个模型在小规模下勉强可用，但随着系统变复杂，它作为一门纪律失败了。一个服务可以在机械意义上是 up 的（响应 health checks），但同时向真实用户返回错误结果、高延迟或间歇性错误。基于已知失败模式的 threshold-based alerting 无法检测新型失败。诊断失败所需的运维知识掌握在特定人类手中，不可复现，也不可转移；当这些人离开时，知识也随之消失。
Google 的 Site Reliability Engineering 职能通常追溯到 2003 年，当时 Ben Treynor Sloss 领导了第一支以软件工程原则处理生产运维的团队。其奠基洞见是：可靠性是工程问题，不是单纯运维问题。如果系统是工程产物，那么系统可靠性就是一种可以被工程化、被量化测量、通过 deliberate design 改善，并像正确性或性能一样被严肃管理的性质。Google 的 SRE teams 由软件工程师组成，而不是由传统运维专家组成；他们被期望通过自动化消除 operational toil，而不是无限期执行它。由此产生的实践——Service Level Indicators、Service Level Objectives、error budgets、blameless postmortems、chaos engineering——都是对工程问题的工程回应，目的是让可靠性工作变得显性、可测量、可改进。
这门纪律随着 Google SRE book 于 2016 年公开化：Beyer、Jones、Petoff 和 Murphy 的 Site Reliability Engineering: How Google Runs Production Systems 免费在线发布。该书第一次把 Google 的 SRE 实践整理成工业界可以直接接触的形式，其影响立即且深远。SRE 作为一种职能、SLOs 作为契约机制、error budgets 作为可靠性—速度权衡工具——这些实践在随后几年广泛传播。Google SRE model 并没有以原始形式被所有组织采纳；多数组织会根据自己的规模和语境改造它。但其中的底层思想——明确量化的可靠性目标、error budgets、系统性 postmortem analysis——改变了工业界思考可靠性的方式。
Observability 维度则以另一条路径、更快速度发展。传统 monitoring——基于 CPU utilization、memory usage、request rates 等 metrics 的 threshold-based alerting——越来越不足以诊断包含许多交互服务的分布式系统中的失败。问题是结构性的：传统 metrics 可以回答聚合系统状态的问题，却无法回答关于单个 requests、users 或 subpopulations 体验的问题。某个特定用户群体可以有很高 error rate，而 overall error rate metrics 看起来正常。某个具体 API endpoint 可以很慢，而 average latency 看起来仍然可接受。
Charity Majors 曾于 2010 年加入 Facebook，后来创立 Honeycomb，她清楚表达了缺失的部分。她的 framing——observability 意味着能够从系统外部输出理解其内部状态，并且适用于系统可能到达的任何状态——指向一种 high-cardinality、high-dimensionality telemetry，这种 telemetry 可以被任意切分和查询，而不只限于预先配置好的维度。带有丰富上下文字段的 structured logs、能够跟随单个 request 穿过 service boundaries 的 distributed traces，以及支持针对任意字段组合进行任意 queries 的 event-driven storage，正是这种能力所需的技术基底。“three pillars of observability” 这一 framing——metrics、logs、traces——成为该领域的组织词汇，后来又随着 continuous profiling 被扩展为第四支柱。
Netflix 的 Chaos Monkey 于 2011 年作为开源工具发布，它会随机终止生产实例，从而形式化了可靠性工程中原本隐含的一种实践：如果你想知道系统如何处理失败，就应该有意识地诱发失败，而不是等它在不方便的时刻自然发生。2016 年发布的 Principles of Chaos Engineering 把这一点编码成一门纪律：定义一个 “steady state” metric，形成关于该 metric 在失败被诱发时如何表现的假设，运行受控实验，并用结果在弱点造成 uncontrolled incidents 之前发现它们。Chaos engineering 从 Netflix-specific practice 转变为更广泛纪律，拥有自己的工具（Gremlin、Chaos Mesh、AWS Fault Injection Simulator）和共同体。
Richard Cook 的 “How Complex Systems Fail” 写于 1998 年，本来面向医疗和航空安全语境，可在线免费获取，却在并非专门讨论软件的情况下，成为这个领域的奠基文本之一。它的十八条原则——复杂系统包含多个 latent failures，失败要求多个 contributing factors 在同一时刻对齐，实践者既是失败的发起者也是最后防线，失败原因总是某次特定条件组合的对齐——都是复杂生产系统的结构性质，任何可靠性工程实践都必须处理它们。Blameless postmortem 成为 SRE 的定义性文化元素之一，正是对 Cook 洞见的应用：proximate human actions 从来不是复杂系统失败的原因；使这些 actions 成为可能的 systemic conditions 才是。
最近最重要的发展，是通过 OpenTelemetry 对 observability instrumentation 的标准化。OpenTelemetry 是 CNCF 项目，于 2019 到 2021 年间达到 production readiness。OpenTelemetry 之前，每个 observability vendor 都要求自己的 SDK，使 instrumentation 与 vendor choice 紧密耦合，并使 vendor switching 成本高到难以承受。OpenTelemetry 提供 vendor-neutral instrumentation，可以把数据发送到任何兼容 backend，把竞争从 instrumentation 转移到 analysis capability。再加上通过 sampling-based approaches，continuous profiling 变得足以在生产环境中持续使用，OpenTelemetry 已经重构了 observability infrastructure landscape，并显著降低了组织建立和维护可观测性的成本。
Service Levels、Failure Analysis 与 Investigation 实践
可靠性目标：SLIs、SLOs 与 Error Budgets
Service Level Indicators（SLIs）测量服务实际正在做什么：error rate、latency distribution、availability、throughput。它们是可靠性对话的量化基础。SLI 的选择很重要：一个服务可以 error rate 很低但 latency 很高；也可以 available 但返回错误结果。SLI 应当测量用户真正关心的东西，而不是容易测量的东西。
Service Level Objectives（SLOs）定义什么级别的 SLI performance 是可接受的。一个 “30 天窗口内 99.9% 的 requests 在 200ms 内成功” 的 SLO，会使可靠性目标显性且可测量。SLO 应设在用户真正需要的可靠性水平——不是 “100% available”，因为这既无法实现也并非必要；也不能低到允许用户体验退化。设定 SLOs 需要对用户对特定失败模式的容忍度有经验性理解。
Error budget 是 SLO 的补集：如果 SLO 要求 99.9% success rate，那么 error budget 就是 0.1%——可接受失败率。Error budgets 是可靠性工程使可靠性—速度权衡显性且可执行的机制。当 error budget 健康时，组织可以承担风险——更激进地部署，尝试新方法。当它被耗尽时，可靠性工作优先于功能开发。这种结构用开发和运维团队预先同意的量化政策，取代了关于“我们是否足够可靠，可以 ship”的隐性协商。
SLO 和 error budget framework 经常被描述为简单，但实践中很难。设定有意义的 SLOs 需要知道用户真正需要什么，这又需要多数团队没有做过的测量和用户研究。强制执行 error budgets 需要组织承诺；而当它阻止团队发布想发布的功能时，开发团队常会抗拒。准确测量 SLIs 需要可能尚不存在的 observability infrastructure。这个框架是一种可靠的工程工具；让它在组织中真正运作，则同时是可靠性工程挑战和组织挑战。
失败的解剖
Cook 的原则捕捉了复杂系统失败的结构事实。没有任何失败只有单一原因。每次失败都是多个 contributing factors 的结果——这些因素单独都不足以造成失败，但在某次特定场合对齐。Proximate cause（失败发生前的直接 action 或 component state）从来不是完整原因；使系统容易受到这个 proximate cause 影响的条件，永远是因果叙事的一部分。
这对 postmortem analysis 有直接含义。一个 postmortem 如果只识别 proximate trigger——“工程师部署了一个配置错误的 rate limiter”——而不识别 contributing conditions——“为什么错误配置能够被部署，为什么它没有在 validation 中失败，为什么 monitoring 没有在影响用户前捕捉到问题，为什么 rate limiter configuration 没有经过 review”——它只是识别了症状，然后停止了。只处理 proximate trigger 的 remediation（培训工程师、增加部署检查）会让 contributing conditions 继续存在，因此类似失败仍可能通过不同 trigger 发生。
Blameless postmortems 在技术上优于 blame-based postmortems，而不只是更友善。Blame 会终止调查：一旦某个人被认定为负责，组织压力就会转向获得道歉并继续前进，而系统性原因还没有被识别出来。Blameless investigation 会对每个 contributing factor 追问：是什么使这个因素成为可能？什么能防止这个因素通过不同机制再次出现在未来事件中？这些答案会产生具体、定向的 remediations，处理系统的 susceptibility，而不是只处理具体 trigger。
Five-whys technique 及其现代衍生方法，是推动 causal analysis 向更深处前进的工具：对于每个已识别原因，继续问是什么导致了这个原因。系统性应用时，这通常会抵达使失败成为可能的系统性条件——组织压力、设计决策、运维实践——而不是停留在人类行动上。该技术的限制在于，它会在一个真实因果结构是 graph 的系统中，产生 linear causal chains；更复杂的方法（fishbone diagrams、fault tree analysis）可以捕捉非线性因果结构。
实践中的可观测性：三支柱及其之后
Metrics 是随时间聚合的数值测量：counters（total requests）、gauges（current memory usage）、histograms（request latency distribution）。它们在高 cardinality 的时间序列上存储和查询效率很高，但会聚合掉每个 request 的细节。Metrics 适合：检测某件事出了问题，测量长期趋势行为，对 threshold violations 发出 alert。它们不适合：理解为什么出问题，理解特定 users 或 requests 的体验，调查 metrics 被定义时未预料到的新型失败模式。
Structured logs 是带有 key-value fields 的单个事件记录，用字段提供上下文：发起 request 的 user，处理它的 service，出现的具体 error，duration。Structured logs 支持事后任意过滤和聚合：“显示过去一小时中 service Y 为 user X 产生的所有 errors” 是 log query，不是 metrics query。它们比 metrics 存储成本更高，但包含调查所需的 per-request detail。从 unstructured logs（用 regex 解析的文本行）转向 structured logs（JSON 或类似格式，带有定义良好的字段），是过去十年中更重要的 observability 改进之一。
Distributed traces 展示单个 request 穿过多个 services 的路径——trace 是一棵 spans 组成的树，每个 span 表示某个 service 中完成的一段工作，并带有 timing、metadata，以及 spans 之间的关系。Distributed tracing 回答的是“这个 request 接触了哪些 services，每个 service 花了多久，每处发生了什么”——这些是 metrics 和 logs 都无法回答的问题，尤其是跨越 service boundaries 的 requests。Dapper paper（Sigelman 等，2010）描述了 Google 内部使用的 tracing system；Zipkin 和 Jaeger 把相同思想实现为开源系统；OpenTelemetry 则标准化了在不同语言和框架中产生 traces 的 instrumentation layer。
Continuous profiling 增加了第四个维度：在某个时间窗口中，哪些代码正在执行，以及执行频率如何。Sampling-based profiling tools（Pyroscope、Parca）以足够低的开销，在生产中持续收集 stack traces，然后聚合并存储，用于回溯分析。与 distributed tracing 结合时，continuous profiling 提供完整图景：requests 正在做什么，以及它们做这些事时，代码如何执行。这弥合了“service 很慢”和“这个具体函数是瓶颈”之间的缺口。
学习这一部分会改变什么
可观测性与可靠性工程会改变实践者在整个软件开发生命周期中思考生产系统的方式，而不只是事故期间。
第一个变化，是在时间压力下进行系统性诊断。生产事故要求在用户受影响、组织迫切要求解决问题的情况下，形成假设，用 telemetry 测试假设，排除替代解释，并收敛到因果解释。这种能力必须通过实践习得——阅读 incident response 并不会直接产生它——但理解假设生成和证据评估的系统方法，可以提供一个框架，使经验能够有效积累。
第二个变化，是从一开始就为可观测性设计。那些在失败期间难以观察的系统，本来就不是以可观测性为需求设计出来的。会发出带有丰富上下文的 structured logs、会在 service calls 之间传播 trace identifiers、会暴露有意义 health endpoints 和 cardinality-appropriate metrics 的代码，反映的是一种设计选择：工程师知道系统最终会出现意外行为，因此预先构建了在这种情况出现时理解它的能力。
第三个变化，是诚实的可靠性记账。设定反映真实用户需要、而不是反映什么容易达成的 SLOs；不软化 metrics 地对其进行测量；在 error budgets 耗尽时强制执行限制——这些都需要让多数团队感到不适的组织纪律。这种不适正是重点：这门纪律使可靠性决策变得显性、可争论，而不是隐性且不可问责；也正因为如此，可靠性才能被系统性改进。
资源
书籍与文本
Beyer、Jones、Petoff 和 Murphy 的 Site Reliability Engineering: How Google Runs Production Systems（O’Reilly，2016，可在 sre.google 免费获取）是经典基础。它定义 SRE 作为一门纪律，覆盖 SLIs、SLOs、error budgets、on-call practices、postmortem culture 和 release engineering，并带有在规模化实践中建立这些方法的实践者权威。部分实践是 Google-specific 的——多数组织没有 Google 的 staffing ratios 或 infrastructure——但底层原则具有普遍迁移性。配套书 Site Reliability Workbook（2018，也可在 sre.google 免费获取）提供案例研究和 worked examples。
Majors、Fong-Jones 和 Miranda 的 Observability Engineering（O’Reilly，2022）是当代可观测性经典文本。它由构建 Honeycomb、并建立现代 observability framing 的实践者写成，覆盖 monitoring 与 observability 的区别，支持真正 observability 的技术基础设施（high-cardinality structured events、trace-based investigation），以及使用它的分析纪律。它是 observability 方面与 SRE books 相配的合适文本。
Forsgren、Humble 和 Kim 的 Accelerate（IT Revolution，2018）为可靠性工程实践提供经验性支持：DORA 研究说明哪些工程实践真正产生可靠、高速度的软件组织。四个关键 metrics（deployment frequency、lead time、mean time to restore、change failure rate）以及这些 metrics 与组织结果之间的研究联系，用证据而非倡议来支持具体可靠性实践。
Rosenthal、Jones、Evans 和 Casey 的 Chaos Engineering（O’Reilly，2020）覆盖 deliberate fault injection 作为一种可靠性实践——chaos engineering 是什么，如何设计实验，它揭示什么，以及如何安全运行它。Principles of Chaos Engineering（principlesofchaos.org，免费）提供这门纪律的经典短表述。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Beyer et al., Site Reliability Engineering（免费） | 经典 SRE 基础 | 入门 |
| Beyer, Murphy et al., The Site Reliability Workbook（免费） | 实践性 SRE 配套 | 深入 |
| Majors, Fong-Jones & Miranda, Observability Engineering | 当代可观测性标准 | 入门 |
| Forsgren, Humble & Kim, Accelerate | DORA 研究；经验性可靠性实践 | 深入 |
| Rosenthal et al., Chaos Engineering | Deliberate fault injection | 深入 |
| Sridharan, Distributed Systems Observability（免费） | 简洁可观测性入门 | 入门 |
| Hollnagel, Resilience Engineering in Practice | 学术 resilience engineering 基础 | 辅助 |
课程、论文与当前资料
Cook 的 “How Complex Systems Fail”（1998，两页，免费在线）是理解复杂系统为何结构性失败的奠基文本。它原本写给医疗和航空安全领域，但完全适用于软件系统。在任何 postmortem 课程或文献之前阅读它，会重塑对整个主题的理解。Allspaw 关于 incident response 的写作——可通过他的博客和 Adaptive Capacity Labs 获取——提供了软件特定的深度，说明工程师在事故中实际如何推理。
Dapper paper（Sigelman 等，Google Technical Report 2010，免费）是 distributed tracing 的奠基论文；理解它可以照亮当代 tracing systems 正在做什么，以及为什么它们以这种方式组织。OpenTelemetry documentation（opentelemetry.io，免费）是 vendor-neutral observability instrumentation 的权威参考。
SREcon talks（USENIX YouTube 免费）代表该领域当前实践者状态。USENIX SREcon conference archives 包含多年 talks，主题包括 reliability practices、incident management、on-call rotation design，以及由大规模运维生产系统的实践者讲解的 observability tooling。
| 资源 | 平台 | 类型 |
|---|---|---|
| Cook, “How Complex Systems Fail”（免费） | 奠基性 failure analysis 文章 | 深入 |
| Allspaw 关于 incident response 的写作（免费） | adaptivecapacitylabs.com | 深入 |
| Sigelman et al., “Dapper” paper（免费） | Distributed tracing 架构论文 | 深入 |
| OpenTelemetry documentation（免费） | opentelemetry.io | 参考 |
| AWS Builders’ Library（免费） | aws.amazon.com/builders-library | 参考 |
| Rob Ewaschuk, “My Philosophy on Alerting”（免费） | linuxczar.net/sysadmin/philosophy-on-alerting | 辅助 |
| Principles of Chaos Engineering（免费） | principlesofchaos.org | 参考 |
| SREcon talks（免费） | USENIX YouTube | 参考 |
| sre.google（SRE books and supporting material） | sre.google | 参考 |
实践、工具与当前资料
公开 postmortems 是可靠性工程中最浓缩的资源之一。Cloudflare、GitHub、Stripe 和 Discord 的 postmortems 通常技术细节充分，写作质量很高。阅读一篇 postmortem，并能够重建其 causal chain、识别 contributing conditions、评估 remediation 是否充分，是一种可靠性工程技能，需要通过真实材料上的练习发展。Cloudflare 的博客存档了多年 postmortems，解释了服务数亿用户的生产系统中的 BGP misconfigurations、software bugs 和 infrastructure failures。
Prometheus 及其生态（Alertmanager、Grafana）是部署最广泛的开源 observability infrastructure。在本地针对一个测试服务运行 Prometheus，定义基于 SLI 的 recording rules，并配置针对 symptoms 而不是 causes 的 alerts，是进入 metrics-based observability 的实践入口。Grafana 的 tutorial documentation 提供了引导式设置。
Jaeger 或 Zipkin（二者都是免费开源）提供 distributed tracing infrastructure。使用 OpenTelemetry 为一个 multi-service application 加 instrumentation，并在 Jaeger 中收集 traces，会使 requests 跨越 service boundaries 的传播过程可见，这种效果不是阅读 tracing 文章可以替代的。
Chaos Monkey 和 Chaos Mesh 开源工具，使受控 failure injection 可用于本地和 cluster environments。运行受控 chaos experiments——随机终止 pods、注入 latency、阻断 network traffic——并观察系统响应，会使 resilience engineering 原则具体化。
| 资源 | 平台 | 类型 |
|---|---|---|
| Cloudflare postmortems and incident reports（免费） | blog.cloudflare.com | 实践 |
| Public postmortems: GitHub、Stripe、Discord（免费） | Various | 实践 |
| Prometheus + Grafana（免费） | prometheus.io / grafana.com | 实践 |
| Jaeger / Zipkin（免费） | jaegertracing.io / zipkin.io | 实践 |
| OpenTelemetry Demo（免费） | opentelemetry.io/docs/demo | 实践 |
| Chaos Monkey / Chaos Mesh（免费） | GitHub / chaos-mesh.org | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 把 SRE 当成 practices checklist | SRE 文献描述了具体实践——SLOs、error budgets、postmortem templates、toil reduction——学习者有时会只在名义上实现它们，而没有底层纪律。结果是组织拥有没有人执行的 SLOs、从未被消耗的 error budgets，以及不会产生改变的 postmortems。这些实践只有在组织承诺支持下才有效；否则就是 cargo cult。 | 将原则与实践一起学习。先理解 error budgets 为什么有效，再实现它们；理解为什么 blameless postmortems 在技术上更优，而不只是文化上更友好；理解为什么 SLO levels 应反映用户需要，而不是当前可达程度。每项实践背后的推理，和实践本身一样重要。 |
| 把 observability 当成 metrics 和 dashboards | Metrics 和 dashboards 是 observability infrastructure 的组成部分，但不是全部。把 observability 理解为“我们有服务 Grafana dashboards”的团队，会在无法用 dashboards 展示的新型失败模式前失去调查能力。即使所有 dashboards 看起来健康，特定用户仍可能经历高 error rates 或 latency，只要 dashboards 不显示 per-user、per-request 或 segment-level behavior。 | 根据 observability infrastructure 能回答什么问题，而不是它产生多少 dashboards 来评估它。投资带有丰富上下文字段的 structured logging，以及支持任意 queries 的 distributed tracing。阅读 Majors 等人的 Observability Engineering，了解真正 observability 的当代标准。 |
| 对一切设置 alert | 认为更多 alerts 能改善检测，因此对大量条件发出 alert，会产生 alert fatigue：pages 太多，工程师无法有效响应，重要 alerts 淹没在噪音中，on-call rotation 变得无法忍受。工程师 silence alerts 并不是不负责任；他们是在理性回应 signal-to-noise 问题。 | 只对直接影响用户、需要人类行动、且人类行动能够处理的 symptoms 发 alert。客户影响不清楚的内部条件，不应产生 pages。SRE book 中关于 alerting philosophy 的章节是合适起点；Rob Ewaschuk 的 “My Philosophy on Alerting” 也提供了很好的补充 framing。 |
| Postmortems 仍然指向责备，即使语气温和 | 如果 postmortems 识别的是失败前接触系统的工程师——哪怕以“我们需要更好培训”或“我们需要更清楚文档”的同情式方式表达——它也会在系统性原因被发现前终止调查。问题不是谁在场，而是什么条件使系统可以通过这种机制失败。无法抵达系统性原因，意味着 remediation 没有处理真正问题。 | 明确围绕 causal systems thinking 训练 postmortem facilitation。对每个已识别 contributing factor，问：是什么使这个因素成为可能？是什么使系统容易受到它影响？什么可以防止这个 contributing factor 通过另一种机制造成类似失败？Cook 的 “How Complex Systems Fail” 是这种 framing 的必要背景。 |
| 把 chaos engineering 视为鲁莽 | 有些组织抵触 chaos engineering，因为它涉及有意在生产系统中制造失败。这种抵触误解了替代方案：生产系统无论如何都会失败，而且会在不可预测的时刻、以不可预测的影响失败。受控 chaos experiments 会在已知条件、已知 blast radius 下暴露失败模式，同时通过定期练习失败来提升 incident response 能力。 | 先在非生产环境中开始 chaos engineering，建立信心；然后在生产中以严格范围限制引入它（canary populations、business hours、显式 rollback mechanisms）。目标不是制造事故，而是在失败模式造成 uncontrolled incidents 之前发现它们。 |
| 不带 SLO 语境地把 MTTF 和 MTTR 当作主要指标 | Mean Time To Failure 和 Mean Time To Recovery 是有用的聚合指标，但它们会掩盖可靠性目标是否真正达成。一个 MTTF 很长、但每次失败都会造成严重用户影响的服务，从用户视角看，可能比一个 MTTF 更短、但恢复迅速且能 graceful degradation 的服务更不可靠。 | 用 user-facing SLIs 和 SLOs 定义可靠性。测量用户体验到的东西，而不是系统内部做了什么。MTTR 可以作为 error budget consumption 的组成部分，但应在其影响的 SLO 语境中解释，而不是作为独立指标解释。 |