English
Reliability engineering asks whether a system delivers the behavior users need, how failures affect them, and how service is restored. A process can be running while requests fail, responses arrive too late, or data is wrong. Start with the user-visible operation rather than equating reliability with machine uptime.
Observability combines instrumentation with investigation. Metrics summarize measurements, logs record events, traces connect work across boundaries, and profiles sample where execution spends resources. Together they help distinguish explanations of unexpected behavior. They do not guarantee that every internal state or cause can be reconstructed: missing context, sampling, collection failures, and ambiguous evidence all matter.
Monitoring and investigation overlap. An alert detects a condition worth action; investigation explains what is happening and tests possible remedies. Site Reliability Engineering (SRE) applies software engineering to operations, including automation, capacity planning, incident response, and reducing repetitive toil. These practices connect service goals to concrete engineering work.
Prerequisites: Operating systems (§4.2) and distributed systems (§4.6). Container orchestration (§4.9) is useful for deployments that use it, but reliability engineering also applies to smaller services and other deployment models.
From Uptime Tracking to Engineering Reliability
Reliability was already a design requirement in early real-time and time-sharing systems. SAGE required continuous operation; interactive users depended on a service being available when they needed it. Redundancy, maintenance, and operational procedures therefore preceded the modern SRE name. Modern practice extends this history rather than inventing systematic reliability engineering from nothing.
As services spread across components, a binary host-up check became insufficient as a description of service quality. A server might answer a health check while one important query was slow or one user population received errors. Request-oriented indicators added evidence about the work actually delivered. Infrastructure measurements remained useful for explaining causes and planning capacity.
Google developed its SRE organization beginning in 2003, connecting software engineering, operations experience, automation, and explicit service objectives. The 2016 book edited by Beyer, Jones, Petoff, and Murphy described these practices for a wider audience. SLIs, SLOs, error budgets, and incident analysis provided a shared way to discuss reliability investment. An SLO is a target; it becomes a contractual commitment only when incorporated into an agreement.
Distributed tracing, including Google’s Dapper work published in 2010, helped connect one request’s work across service boundaries. Structured events and richer query systems also supported investigating subsets hidden by an aggregate average. Work by practitioners including Charity Majors helped popularize this investigative emphasis. Useful telemetry still depends on context propagation, meaningful fields, and adequate coverage.
Netflix’s Chaos Monkey and later chaos-engineering practice made deliberate failure experiments more widely known. The useful method is to state a hypothesis, observe a relevant service indicator, introduce a bounded failure, and compare the outcome. Random disruption without a question and a recovery plan does not by itself establish resilience.
Richard Cook’s 1998 “How Complex Systems Fail” encouraged analysis of interacting conditions, adaptations, and defenses. Blameless incident reviews apply a related approach: examine the information available to participants and how the system permitted the outcome. This does not mean human actions never contribute, or that accountability disappears. It means identifying a person is insufficient as an explanation or a repair.
OpenTelemetry developed common interfaces and formats for collecting, processing, and exporting telemetry. It can reduce coupling between application instrumentation and a particular backend. Support varies by signal, language, and component; adopting one library does not automatically provide complete traces or useful service objectives. The enduring contribution is interoperability at defined interfaces, not the elimination of instrumentation design.
Service Levels, Failure Analysis, and the Practice of Investigation
Connect a user-visible requirement to a measurement, a target, and a response when the target is threatened. Investigation then links a deviation to evidence about components, changes, and workload.
Reliability Targets: SLIs, SLOs, and Error Budgets
An SLI is a defined measurement, such as the fraction of eligible checkout requests that succeed within 200 ms. Specify the population, observation point, time window, and treatment of missing data. A server-side success counter can miss requests that never reached the server; a successful HTTP status can still accompany an incorrect business result.
An SLO states a target for that measurement, such as at least 99.9% good requests over 30 days. An SLA is an agreement with stated commitments and consequences; it need not equal the internal SLO. Choose a target that reflects user needs and engineering cost. Calling something “100%” does not establish a guarantee, and a brief observation of zero failures does not predict indefinite perfection.
For ten million eligible requests and a 99.9% target, the error budget is 10,000 bad requests. A time-based target of 99.9% over 30 days instead permits 43.2 minutes of bad time. These denominators are different: a request-based budget does not turn directly into outage minutes when traffic varies. Separate objectives may be needed for different operations or user groups.
Burn rate compares the observed bad-event fraction with the allowed fraction. A 1% error rate against a 0.1% allowance is a burn rate of ten. For request-based objectives, actual budget consumption over elapsed time also depends on traffic volume. The SRE workbook uses multiple observation windows to distinguish sustained budget consumption from brief noise.
An error-budget policy defines what action follows: investigate, reduce rollout risk, prioritize a repair, or pause a class of changes. A blanket deployment freeze is one possible policy, not a mathematical consequence of the SLO, and a necessary reliability fix may itself require deployment. Agree on the response before an incident makes the decision urgent.
The Anatomy of Failures
An incident can have a trigger, conditions that allow it to spread, and failed or successful defenses. For example, a database slowdown increases request duration; clients retry; retries increase load; queues grow further. Each local retry may appear reasonable while the combined behavior overloads the dependency. Deadlines, bounded retries with backoff and jitter, admission control, and isolation of resource pools address different parts of that feedback.
Build a timeline from evidence, distinguish observations from hypotheses, and ask what would have changed the outcome. A deployment may precede an outage without causing it. A useful repair targets a supported mechanism: validating a configuration before use, limiting fan-out, making rollback possible, or reducing a dependency’s load. Assign an owner and check whether the repair works.
A blameless review asks why an action made sense with the information and tools available at the time. It should still record decisions and responsibilities. “Human error” is too broad to tell you what to improve, just as “distributed systems are hard” is too broad. Five-whys questions can start investigation, but they can also force a complex event into one unsupported chain; retain competing explanations until evidence distinguishes them.
Recovery deserves design and practice. Backups need restoration tests, failover needs capacity, and redundancy needs independence from the failure being considered. Two replicas behind one failed network dependency may both be unreachable. Reliability improves when the whole recovery path works, not merely when the component count increases.
Observability in Practice: The Three Pillars and Beyond
Metrics summarize quantities over time: counters record accumulated events, gauges record current values, and histograms retain a binned distribution. They can help both detection and diagnosis, provided the chosen dimensions distinguish the cases that matter. High-cardinality labels create many distinct time series; adding a user identifier to every metric can be expensive. Choose dimensions according to the question and storage model.
Averages can conceal a slow minority. A p99 latency identifies a threshold at or below which about 99% of the measured requests fall, under the estimator used. Averaging several instances’ p99 values does not yield the whole service’s p99. Aggregate compatible histogram counts or the underlying observations when a combined distribution is required, and keep the measurement window and population explicit.
Structured logs preserve event context and can be linked to traces by identifiers. Record useful fields with a stable meaning, while controlling access and avoiding unnecessary secrets or personal data. Logs still omit anything not recorded, can be dropped, and may be sampled; a missing line alone does not prove that an action never occurred.
Traces connect spans of work through propagated context. Parent-child relationships and span links can represent synchronous calls and asynchronous relationships. Missing propagation breaks connections, and sampling can omit the rare request you need. OpenTelemetry’s signal model supplies common interfaces; useful investigation also requires consistent identifiers, timestamps, and application meaning.
Profiles sample execution or allocation to locate resource costs. A CPU profile does not directly measure all time spent waiting for I/O or locks, and instrumentation has overhead. Combine signals around a hypothesis: for a slow endpoint, compare its latency distribution, trace waits, queue depth, and relevant profiles. Correlated changes suggest where to investigate, while controlled comparisons help establish a cause.
What Studying This Changes
Observability and reliability engineering changes how practitioners think about production systems throughout the software development lifecycle, not only during incidents.
The first change is systematic diagnosis under time pressure. Production incidents require forming hypotheses, testing them against telemetry, eliminating alternatives, and converging on a causal explanation while users are affected and organizational pressure to resolve the incident is acute. This capability is learned by doing — reading about incident response does not produce it — but understanding the systematic approach to hypothesis generation and evidence evaluation provides the framework within which experience accumulates productively.
Design instrumentation around questions that arise during operation: which requests failed, which dependency delayed them, and which change altered behavior? Propagated identifiers, meaningful latency distributions, and relevant events preserve evidence for those questions. Revisit coverage after incidents expose missing information, and measure the instrumentation’s own reliability and cost.
Reliability accounting makes priorities reviewable. Preserve the original SLI definition when performance deteriorates, report which users and operations were affected, and connect follow-up work to the observed failure. Revisit a target when requirements change, with an explicit explanation, rather than quietly redefining success to hide a miss.
Resources
Books and Texts
Site Reliability Engineering and The Site Reliability Workbook explain service objectives, operations, and budget policies. Observability Engineering develops investigative use of telemetry; Distributed Systems Observability offers a shorter entry. Accelerate examines organizational delivery practices, while chaos and resilience engineering texts explore experimentation and adaptation. Read empirical associations with their study design rather than treating them as guaranteed causal effects for every team.
| Book | Role | Type |
|---|---|---|
| Beyer et al., Site Reliability Engineering (free) | Service objectives and operational engineering | Entry |
| Beyer, Murphy et al., The Site Reliability Workbook (free) | Practical SRE companion | Depth |
| Majors, Fong-Jones & Miranda, Observability Engineering | Telemetry and production investigation | Entry |
| Forsgren, Humble & Kim, Accelerate | DORA research; empirical reliability practices | Depth |
| Rosenthal et al., Chaos Engineering | Deliberate fault injection | Depth |
| Sridharan, Distributed Systems Observability (free) | Concise observability introduction | Entry |
| Hollnagel, Resilience Engineering in Practice | Academic resilience engineering foundations | Auxiliary |
Courses, Papers, and Current Sources
Cook’s How Complex Systems Fail and Allspaw’s incident-analysis writings encourage examining conditions and adaptations. The Dapper paper describes distributed tracing, while OpenTelemetry specifies instrumentation interfaces. SREcon talks, the AWS Builders’ Library, and alerting essays provide operational cases. For each case, identify the evidence, explanation, intervention, and limits of applying it elsewhere.
| Resource | Platform | Type |
|---|---|---|
| Cook, “How Complex Systems Fail” (free) | Foundational failure analysis essay | Depth |
| Allspaw’s writings on incident response (free) | adaptivecapacitylabs.com | Depth |
| Sigelman et al., “Dapper” paper (free) | Distributed tracing architecture paper | Depth |
| OpenTelemetry documentation (free) | opentelemetry.io | Reference |
| AWS Builders’ Library (free) | aws.amazon.com/builders-library | Reference |
| Rob Ewaschuk, “My Philosophy on Alerting” (free) | linuxczar.net/sysadmin/philosophy-on-alerting | Auxiliary |
| Principles of Chaos Engineering (free) | principlesofchaos.org | Reference |
| SREcon talks (free) | USENIX YouTube | Reference |
| sre.google (SRE books and supporting material) | sre.google | Reference |
Practice, Tools, and Current Sources
Instrument a small service with an SLI, a latency histogram, structured events, and propagated trace context. Prometheus, Grafana, Jaeger, Zipkin, and the OpenTelemetry Demo provide different practice components. Introduce a controlled delay and test whether the telemetry distinguishes dependency waiting from CPU work. Read a public incident report and evaluate whether its proposed repairs address the described mechanism; use fault-injection tools only as part of a defined experiment.
| Resource | Platform | Type |
|---|---|---|
| Cloudflare postmortems and incident reports (free) | blog.cloudflare.com | Practice |
| Public postmortems: GitHub, Stripe, Discord (free) | Various | Practice |
| Prometheus + Grafana (free) | prometheus.io / grafana.com | Practice |
| Jaeger / Zipkin (free) | jaegertracing.io / zipkin.io | Practice |
| OpenTelemetry Demo (free) | opentelemetry.io/docs/demo | Practice |
| Chaos Monkey / Chaos Mesh (free) | GitHub / chaos-mesh.org | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Adopting SRE practices only by name | A target without measurement or an agreed response cannot guide decisions. | Connect the SLI, SLO, budget policy, and owned follow-up work. |
| Equating dashboards with complete observability | A dashboard answers selected questions and may hide affected subpopulations. | Test whether available telemetry can answer a concrete incident question. |
| Paging on every anomaly | Noise makes urgent signals harder to recognize. | Page for actionable urgency, including imminent impact; route lower-priority findings to other workflows. |
| Stopping at who made the last change | Temporal proximity alone does not explain a failure. | Reconstruct the mechanism and contributing conditions, then verify the proposed repair. |
| Running failure experiments without a hypothesis | Disruption alone does not measure resilience. | State the expected behavior, bound the experiment, and observe recovery against it. |
| Using one recovery average as the whole reliability measure | A mean hides impact, variation, and which operations failed. | Combine recovery measurements with user-facing indicators and incident severity. |
中文
可靠性工程询问系统是否提供用户所需行为、故障怎样影响用户，以及怎样恢复服务。进程仍在运行时，请求也可能失败、响应过晚或数据错误。应从用户可见操作开始，而不是把可靠性等同于机器在线时间。
可观测性结合埋点与调查。指标概括测量，日志记录事件，追踪连接跨边界工作，性能剖析抽样观察执行把资源用在哪里。它们共同帮助区分异常行为的不同解释，但不保证能够重建每个内部状态或原因；缺失上下文、采样、采集失败与证据歧义都会产生影响。
监控与调查相互重叠。告警检测值得采取行动的条件，调查则解释正在发生什么，并检验可能的补救措施。站点可靠性工程（SRE）把软件工程用于运维，包括自动化、容量规划、事件响应与减少重复劳动。这些实践把服务目标连接到具体工程工作。
前置知识：操作系统（§4.2）与分布式系统（§4.6）。对于使用容器编排的部署，§4.9 很有帮助；但可靠性工程也适用于较小服务及其他部署方式。
从 Uptime Tracking 到工程化可靠性
可靠性在早期实时系统与分时系统中已经是设计要求。SAGE 需要连续运行，交互式用户也依赖服务在需要时可用。因此，冗余、维护与运行程序早于现代 SRE 名称。现代实践延续了这段历史，并非从无到有发明了系统性的可靠性工程。
服务分散到多个组件后，主机在线或离线的二元检查，不足以描述服务质量。服务器可以通过健康检查，但某个重要查询很慢，或某类用户持续收到错误。面向请求的指标补充了实际交付工作的证据。基础设施测量仍然有助于解释原因和规划容量。
Google 从 2003 年开始发展 SRE 组织，连接软件工程、运维经验、自动化与明确服务目标。Beyer、Jones、Petoff 与 Murphy 主编的 2016 年书籍，向更广泛读者介绍了这些实践。SLI、SLO、错误预算与事件分析，为讨论可靠性投入提供共同依据。SLO 是目标，只有被纳入协议时，才成为合同承诺。
分布式追踪，包括 Google 在 2010 年发表的 Dapper 工作，帮助连接一个请求在多个服务边界上的工作。结构化事件与更丰富的查询系统，也支持调查被总体平均值掩盖的子集。包括 Charity Majors 在内的实践者，帮助推广了这种重视调查的思路。有用的遥测仍依赖上下文传播、有意义的字段与足够覆盖。
Netflix 的 Chaos Monkey 与后来的混沌工程实践，使主动故障实验更加普及。有用的方法是提出假设，观察相关服务指标，引入有界故障，再比较结果。没有问题与恢复计划的随机扰动，本身不能建立韧性。
Richard Cook 在 1998 年的《How Complex Systems Fail》中，鼓励分析相互作用的条件、适应行为与防御措施。无责事件复盘采用相关方法：检查参与者当时掌握的信息，以及系统怎样允许结果发生。这不表示人的行为从不产生影响，也不表示责任消失，而是说指出某个人不足以构成解释或修复。
OpenTelemetry 发展了采集、处理与导出遥测的公共接口与格式，可以减少应用埋点与特定后端的耦合。支持情况依信号、语言与组件而异；采用一个库不会自动提供完整追踪或有用的服务目标。持续有用的贡献，是明确接口上的互操作性，而不是消除埋点设计。
Service Levels、Failure Analysis 与 Investigation 实践
把用户可见要求连接到测量、目标，以及目标受到威胁时的应对方式。调查再把偏差连接到组件、变更与工作负载的证据。
可靠性目标：SLIs、SLOs 与 Error Budgets
SLI 是一个明确定义的测量，例如符合统计条件的结账请求中，200 毫秒内成功的比例。需要规定统计总体、观察位置、时间窗口与缺失数据处理方式。服务器端成功计数可能漏掉根本没到达服务器的请求；成功的 HTTP 状态也可能伴随错误业务结果。
SLO 为这个测量设定目标，例如 30 天内至少 99.9% 的请求符合要求。SLA 则是规定承诺与后果的协议，不必等同于内部 SLO。目标应反映用户需求与工程成本。把某件事称为“100%”不会建立保证，短时间内没有观察到失败，也不能预测无限期完美运行。
对于一千万个符合统计条件的请求和 99.9% 的目标，错误预算为 10,000 个不合格请求。若采用基于时间的 30 天 99.9% 目标，则允许 43.2 分钟不合格时间。这是不同分母：流量变化时，不能直接把请求预算换算成停机分钟。不同操作或用户群也可能需要分别设定目标。
燃烧率把观察到的不合格事件比例与允许比例比较。允许 0.1% 错误、实际为 1% 时，燃烧率为十。对于基于请求的目标，某段时间实际消耗多少预算，还依赖流量。SRE 工作手册使用多个观察窗口，区分持续预算消耗与短暂噪声。
错误预算策略规定后续行动：调查、降低发布风险、优先修复，或暂停某类变更。全面冻结发布是一种可能策略，并非 SLO 的数学必然结果；必要的可靠性修复本身也可能需要部署。应在事件使决策紧迫之前，商定应对方式。
失败的解剖
一次事件可以包含触发因素、允许影响扩大的条件，以及成功或失败的防御。例如，数据库变慢使请求持续时间增加，客户端开始重试，重试提高负载，队列进一步增长。每次本地重试看似合理，组合起来却使依赖过载。截止时间、带退避与随机扰动的有界重试、准入控制，以及资源池隔离，分别处理这种反馈中的不同部分。
依据证据建立时间线，区分观察与假设，并询问什么会改变结果。部署发生在故障之前，不代表部署造成故障。有用的修复针对已有证据支持的机制，例如使用前验证配置、限制扇出、使回退可行，或降低依赖负载。应指定负责人，并检查修复是否有效。
无责复盘询问：在当时的信息与工具条件下，为什么某个行动看起来合理。它仍然应记录决策与职责。“人为错误”过于笼统，不能告诉你该改什么，正如“分布式系统很难”也不够。连续追问为什么可以启动调查，但也可能把复杂事件压成一条缺乏依据的链；应保留竞争解释，直到证据能区分它们。
恢复也需要设计与演练。备份需要还原测试，故障转移需要容量，冗余则需要独立于所考虑的故障。两个副本如果依赖同一个失败的网络组件，可能同时不可达。可靠性提升需要整个恢复路径有效，而不只是增加组件数量。
实践中的可观测性：三支柱及其之后
指标概括随时间变化的量：计数器记录累计事件，仪表记录当前值，直方图保留分桶分布。只要选定维度能区分重要情况，指标就既能帮助检测，也能帮助诊断。高基数标签产生大量不同时间序列；给每个指标增加用户标识可能很昂贵。应按问题与存储模型选择维度。
平均值可以掩盖较慢的少数请求。p99 延迟给出一个阈值，按所使用的估计方法，约 99% 的被测请求延迟不高于该值。把多个实例的 p99 求平均，得不到整个服务的 p99。需要总体分布时，应聚合兼容的直方图计数或原始观察，并明确测量窗口与总体。
结构化日志保留事件上下文，可以通过标识与追踪关联。应记录含义稳定、有用的字段，同时控制访问，避免不必要的秘密信息或个人数据。日志仍会遗漏未记录内容，可能丢失，也可能被采样；缺少一行日志，本身不能证明某个动作从未发生。
追踪通过传播的上下文连接工作片段。父子关系与 span link 可以表示同步调用与异步关联。缺少传播会中断连接，采样也可能漏掉需要调查的罕见请求。OpenTelemetry 的信号模型提供公共接口；有效调查还需要一致的标识、时间戳与应用含义。
性能剖析抽样执行或分配，以定位资源成本。CPU 剖析不会直接测量等待 I/O 或锁的全部时间，埋点也有开销。应围绕假设组合信号：对于慢端点，比较延迟分布、追踪中的等待、队列深度与相关剖析。同步变化提示调查方向，受控比较则帮助建立因果解释。
学习这一部分会改变什么
可观测性与可靠性工程会改变实践者在整个软件开发生命周期中思考生产系统的方式，而不只是事故期间。
第一个变化，是在时间压力下进行系统性诊断。生产事故要求在用户受影响、组织迫切要求解决问题的情况下，形成假设，用 telemetry 测试假设，排除替代解释，并收敛到因果解释。这种能力必须通过实践习得——阅读 incident response 并不会直接产生它——但理解假设生成和证据评估的系统方法，可以提供一个框架，使经验能够有效积累。
围绕运行期间的问题设计埋点：哪些请求失败，哪个依赖延迟了请求，以及哪项变更改变了行为？传播的标识、有意义的延迟分布与相关事件，为这些问题保留证据。事件暴露缺失信息后，应重新检查覆盖，也测量埋点自身的可靠性与成本。
可靠性核算使优先级可以被审查。性能恶化时，应保留原来的 SLI 定义，报告哪些用户和操作受到影响，并把后续工作连接到观察到的故障。需求变化时可以重新评估目标，但应明确解释，不能悄悄重定义成功来掩盖未达标。
资源
书籍与文本
Site Reliability Engineering 与 The Site Reliability Workbook 解释服务目标、运行与预算策略。Observability Engineering 展开利用遥测开展调查的方法，Distributed Systems Observability 提供较短的入门。Accelerate 研究组织交付实践，混沌工程与韧性工程文本则探索实验和适应。阅读经验关联时，应考虑研究设计，不能把它们当作对每个团队都保证成立的因果效果。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Beyer et al., Site Reliability Engineering（免费） | 服务目标与运行工程 | 入门 |
| Beyer, Murphy et al., The Site Reliability Workbook（免费） | 实践性 SRE 配套 | 深入 |
| Majors, Fong-Jones & Miranda, Observability Engineering | 遥测与生产环境调查 | 入门 |
| Forsgren, Humble & Kim, Accelerate | DORA 研究；经验性可靠性实践 | 深入 |
| Rosenthal et al., Chaos Engineering | Deliberate fault injection | 深入 |
| Sridharan, Distributed Systems Observability（免费） | 简洁可观测性入门 | 入门 |
| Hollnagel, Resilience Engineering in Practice | 学术 resilience engineering 基础 | 辅助 |
课程、论文与当前资料
Cook 的 How Complex Systems Fail 与 Allspaw 的事件分析文章，鼓励检查条件与适应行为。Dapper 论文 描述分布式追踪，OpenTelemetry 规定埋点接口。SREcon 演讲、AWS Builders’ Library 与告警文章提供运行案例。每个案例都应识别证据、解释、干预，以及迁移到其他环境的限制。
| 资源 | 平台 | 类型 |
|---|---|---|
| Cook, “How Complex Systems Fail”（免费） | 奠基性 failure analysis 文章 | 深入 |
| Allspaw 关于 incident response 的写作（免费） | adaptivecapacitylabs.com | 深入 |
| Sigelman et al., “Dapper” paper（免费） | Distributed tracing 架构论文 | 深入 |
| OpenTelemetry documentation（免费） | opentelemetry.io | 参考 |
| AWS Builders’ Library（免费） | aws.amazon.com/builders-library | 参考 |
| Rob Ewaschuk, “My Philosophy on Alerting”（免费） | linuxczar.net/sysadmin/philosophy-on-alerting | 辅助 |
| Principles of Chaos Engineering（免费） | principlesofchaos.org | 参考 |
| SREcon talks（免费） | USENIX YouTube | 参考 |
| sre.google（SRE books and supporting material） | sre.google | 参考 |
实践、工具与当前资料
给小型服务增加一个 SLI、延迟直方图、结构化事件与传播的追踪上下文。Prometheus、Grafana、Jaeger、Zipkin 与 OpenTelemetry Demo 提供不同练习组件。引入受控延迟，检查遥测能否区分等待依赖与 CPU 工作。阅读一份公开事件报告，评估所提修复是否针对描述中的机制；故障注入工具应服务于明确实验。
| 资源 | 平台 | 类型 |
|---|---|---|
| Cloudflare postmortems and incident reports（免费） | blog.cloudflare.com | 实践 |
| Public postmortems: GitHub、Stripe、Discord（免费） | Various | 实践 |
| Prometheus + Grafana（免费） | prometheus.io / grafana.com | 实践 |
| Jaeger / Zipkin（免费） | jaegertracing.io / zipkin.io | 实践 |
| OpenTelemetry Demo（免费） | opentelemetry.io/docs/demo | 实践 |
| Chaos Monkey / Chaos Mesh（免费） | GitHub / chaos-mesh.org | 实践 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 只在名称上采用 SRE 实践 | 没有测量与约定应对的目标，不能指导决策。 | 连接 SLI、SLO、预算策略与有负责人的后续工作。 |
| 把仪表板等同于完整可观测性 | 仪表板回答选定问题，可能掩盖受影响子群。 | 检验现有遥测能否回答一个具体事件问题。 |
| 对每种异常都紧急呼叫 | 噪声使紧急信号更难识别。 | 对需要紧急行动的情况呼叫，包括迫近的影响；较低优先级发现进入其他流程。 |
| 只追问谁作了最后变更 | 时间接近本身不能解释故障。 | 重建机制与促成条件，再验证所提修复。 |
| 没有假设就进行故障实验 | 只有扰动不能衡量韧性。 | 说明预期行为，限制实验范围，并据此观察恢复。 |
| 用一个平均恢复时间代表全部可靠性 | 平均值掩盖影响、差异与哪些操作失败。 | 把恢复测量与用户可见指标、事件严重程度结合。 |