English
DevOps is not a role, a tool, or a methodology with a fixed definition. It is a cultural and technical movement that emerged in response to a specific problem: the structural antagonism between development teams (whose goal is to change software) and operations teams (whose goal is to keep software stable). When these goals are held by different groups with different incentives, different tools, and different knowledge, the result is slow delivery, frequent failures, and organizations where software changes are treated as emergencies rather than routine events. DevOps is the attempt to resolve this antagonism by giving development teams ownership of what happens in production and giving operations teams the means to make production changes safe.
The practices that constitute DevOps converge on a common mechanism: automating everything that can be automated, so that human judgment is reserved for decisions that actually require it, and so that the path from code to production is reliable enough that frequent changes are less risky than infrequent large ones. A deployment that requires a manual checklist of forty steps, executed by a specialized team on a quarterly schedule, is a deployment that will have errors, that teams will avoid making, and that will accumulate large batches of change that make failures hard to diagnose. A deployment that is fully automated, executed by any developer on demand, and monitored automatically is a deployment that teams make routinely, that produces small batches of change, and whose failures are isolated and diagnosable.
Continuous delivery — the practice of keeping software in a releasable state at all times, deployable to production on demand — is the technical expression of what DevOps aims to achieve organizationally. It requires an automated deployment pipeline that builds, tests, and deploys every code change through a sequence of environments; tests that run quickly enough to provide feedback within minutes; and the organizational discipline to prioritize pipeline health over feature delivery. Continuous deployment, the stricter practice, deploys every change to production automatically on passing all pipeline stages without human approval.
Prerequisites: Testing (§7.2) — automated tests are what make fast deployment safe. Software process (§7.3) — DevOps is a process practice with organizational implications. Container orchestration (§4.9) — modern deployment infrastructure. Observability and reliability engineering (§4.10) — production monitoring is central to DevOps practice.
From Wall of Confusion to Platform Engineering
The problem that DevOps addresses was created by a organizational structure that made intuitive sense at the time it was established.
In the 1990s and early 2000s, it was common for organizations to separate software development from operations. Development teams wrote the software; operations teams deployed and maintained it. This separation reflected real concerns: developers making ad hoc changes to production systems could cause outages; a specialized operations team with controlled access could maintain stability. The separation also reflected organizational logic: the skills for writing code and the skills for managing servers were different, and grouping people by skill was natural.
The problem was structural. Development teams were measured on shipping features; operations teams were measured on uptime. Feature shipping requires change; uptime requires stability. The incentives were in direct conflict, and the conflict expressed itself as a wall: development threw software over the wall to operations, operations threw it back when it failed to deploy, and both teams blamed each other. Deployments became infrequent, planned events — quarterly or less — because each deployment required coordination between multiple teams with conflicting priorities, extensive manual verification, and scheduled maintenance windows. Infrequent deployments accumulated large batches of change, which made failures impossible to isolate and diagnose. The wall of confusion, as it became known in the DevOps community, made both development and operations worse at their stated goals.
Patrick Debois, a Belgian consultant who had been studying ways to bridge the gap between development and operations, used the term “DevOps” in 2009 when organizing the first DevOpsDays conference in Ghent. The name stuck. The movement grew rapidly because it addressed a real problem that many organizations recognized in themselves. The Agile movement had improved development practices but had not addressed the handoff between development and operations; DevOps completed the circuit.
Gene Kim, Kevin Behr, and George Spafford’s The Phoenix Project (2013) brought DevOps to a much wider audience through a novel format: a story about an IT organization in crisis, following the protagonist Bill Palmer as he discovers and implements the Three Ways of DevOps. The Three Ways — Flow (fast, smooth delivery of work), Feedback (seeing and solving problems quickly), and Continual Learning (safe experimentation and learning from success and failure) — provided a framework for understanding why specific DevOps practices work. The Phoenix Project was arguably more influential than any technical book because it reached managers and executives who would never read a technical implementation guide.
The infrastructure-as-code movement paralleled the DevOps movement and was essential to it. If infrastructure is managed by running commands on servers, there is no audit trail, no repeatability, and no way to recover from failure without human memory. If infrastructure is defined as code — in configuration management tools like Puppet (2005), Chef (2009), and Ansible (2012), or in declarative provisioning tools like Terraform (2014) — it becomes testable, versionable, and reproducible. The same code that provisions a development environment provisions the production environment; differences between environments are visible in code diffs rather than in mysterious behavioral discrepancies.
Docker’s introduction in 2013 changed the operational model by making containers — lightweight, portable, reproducible execution environments — practical for application deployment. Before Docker, deployments were complex because the same application behaved differently on different servers due to differences in installed packages, configuration files, and library versions. A Docker container packages the application and all its dependencies in a single artifact that runs identically wherever it is deployed. This eliminated the class of bugs that came from environment inconsistency and made the separation of “build once, run anywhere” achievable. Combined with Kubernetes for orchestration (§4.9), Docker containers became the dominant deployment artifact for cloud-native applications.
The DORA research program (§7.3) provided the empirical validation that gave the DevOps movement credibility beyond narrative and intuition. Its finding that elite performers deployed on-demand with low change failure rates — and that this combination predicted organizational performance, not just delivery performance — was the evidence that DevOps practices were not merely developer preferences but business differentiators. The technical practices it identified as predictive of elite performance — trunk-based development, continuous integration, deployment automation, monitoring and observability, test automation, database change management — became the canonical list of what DevOps required technically.
Google’s Site Reliability Engineering model (SRE, §4.10) addressed a related but distinct problem: how to keep large, complex systems running reliably at scale. SRE defines reliability targets through Service Level Objectives, uses error budgets to balance reliability and velocity, treats toil (manual operational work that can be automated) as an enemy to eliminate, and expects SRE engineers to spend a significant fraction of their time writing code to automate operations rather than performing operations manually. SRE and DevOps are complementary: DevOps addresses the organizational structure between development and operations; SRE addresses the practices within operations for high-reliability systems.
Platform engineering emerged in the late 2010s and early 2020s as the maturation of the DevOps movement. As organizations adopted DevOps practices, they discovered that asking every development team to independently manage their deployment pipelines, observability infrastructure, and cloud provisioning produced inconsistency and duplicated effort. Platform engineering teams build internal developer platforms — the tooling, APIs, and abstractions that make doing the right thing the easy thing for development teams. The Internal Developer Platform provides golden paths for common tasks (deploying a new service, setting up observability, managing secrets) that implement best practices without requiring each team to discover those practices independently. The Team Topologies framework (§7.3) classifies the platform engineering team as a platform team, providing X-as-a-Service to stream-aligned development teams.
The Deployment Pipeline, Infrastructure as Code, and Observability
The Deployment Pipeline as Organizational Backbone
The deployment pipeline is the automated sequence of stages that every code change passes through on its way from developer commit to production deployment. Its structure reflects the principle that feedback should come as early and as cheaply as possible: faster, cheaper stages run first; slower, more expensive stages run later.
A canonical pipeline has several stages. The commit stage — running immediately on every code push — compiles the code, runs unit tests, performs static analysis, and produces a deployable artifact. This stage should complete in five minutes or less; if it is slow, developers cannot get fast feedback and will work in larger batches. The acceptance stage runs integration tests and automated acceptance tests against the deployed artifact in a production-like environment; it provides the verification that the system works together as expected. Performance tests and security scans may run in parallel or subsequent stages. The production deployment stage deploys to production — either automatically on passing all previous stages (continuous deployment) or with a human approval gate (continuous delivery).
The pipeline requires several supporting practices to work. Artifact promotion — using the same artifact (the same container image or compiled binary) through all pipeline stages, rather than rebuilding in each stage — ensures that what is tested is what is deployed. Environment parity — making development, staging, and production environments as similar as possible — ensures that tests in staging catch problems that will appear in production. Pipeline health monitoring — treating a failing pipeline stage with the urgency of a production incident — ensures that the pipeline provides reliable feedback rather than becoming a noise source that teams learn to ignore.
Feature flags (also called feature toggles) allow code to be deployed to production before the feature is enabled for users. The flag separates deployment from release: the code is in production, but the feature is not yet visible. This enables trunk-based development (all changes merge to main, which is always deployable) even for features that require weeks of development. It enables gradual rollout — enabling a feature for 1% of users, then 10%, then 100% — which reduces the blast radius of bugs. It enables A/B testing — different users get different versions of the feature — which enables empirical optimization. Feature flags require management: flags that are no longer needed should be removed, and a large number of active flags creates combinatorial complexity in the code.
Blue-green deployment and canary releases are strategies for reducing the risk of production deployments. In blue-green deployment, two production environments (blue and green) exist simultaneously; traffic shifts from one to the other on deployment, and rollback is instantaneous by shifting traffic back. In canary release, a small percentage of production traffic (say 1%) is routed to the new version while the rest continues to the old version; metrics are monitored, and the percentage increases only if the new version performs as expected. Both strategies allow early detection of production problems before they affect all users.
Infrastructure as Code and Configuration Management
Infrastructure as code (IaC) treats the provisioning and configuration of infrastructure — servers, networks, databases, load balancers — as software, managed with the same tools (version control, code review, automated testing) as application code. The benefits are repeatability (the same configuration applied produces the same result), auditability (every change is a commit with an author and a message), and disaster recovery (infrastructure can be reproduced from code rather than from memory).
Terraform (HashiCorp, 2014) is the dominant tool for declarative infrastructure provisioning. A Terraform configuration describes the desired state of cloud infrastructure — which virtual machines, which networking, which managed services — and Terraform computes the changes needed to bring the actual state in line with the desired state. The plan/apply workflow — compute the changes, review them, apply them — provides visibility into what changes will be made before they are made. Terraform state — the record of what Terraform believes is currently provisioned — is the critical artifact that must be managed carefully; state corruption or inconsistency produces unpredictable behavior.
Ansible, Puppet, and Chef address configuration management — the ongoing management of software installed on servers and how it is configured. These tools are less central in cloud-native environments where immutable infrastructure (servers are replaced rather than updated) is the deployment model, but they remain important for managing systems that predate containerization or that cannot be containerized.
GitOps, a practice that emerged from the Kubernetes ecosystem, uses a Git repository as the source of truth for cluster configuration. A GitOps controller (Argo CD, Flux) continuously compares the desired state described in the repository with the actual state of the cluster and applies changes to bring the cluster in line. Every change to infrastructure is a pull request, with review, approval, and audit trail. Rollback is a revert commit. GitOps is the cleanest implementation of infrastructure as code for Kubernetes deployments.
Measuring and Monitoring Production
Observability — the ability to understand the internal state of a production system from its external outputs — is what makes frequent deployment safe. If a team cannot see what their software is doing in production, they cannot know whether a deployment has succeeded, whether a change in behavior is a regression, or whether users are experiencing problems. Without observability, frequent deployment is reckless; with it, frequent deployment is risk management.
The DORA research identifies monitoring and observability as one of the technical practices most strongly predictive of software delivery performance. Teams with good observability detect and diagnose problems faster, which allows smaller error budgets to be spent on learning rather than on extended outages. The specific practices — structured logging, metrics with meaningful cardinality, distributed tracing, and increasingly continuous profiling — are described in §4.10.
The deployment frequency measurement is the simplest observability signal for the deployment pipeline itself. Teams that track how often they deploy to production and that can see the trend over time have the most basic feedback loop on process health. A team that deploys once per month and wants to deploy daily has a clear, measurable goal; a team that does not track deployment frequency cannot know whether it is improving.
What Studying This Changes
DevOps changes how practitioners think about the relationship between software development and production operation.
The first change is the ability to identify the cost of the wall of confusion. Organizations where development and operations are separate teams with separate incentives produce the characteristic symptoms: infrequent deployments, high change failure rates, long recovery times. Practitioners who understand the structural source of these symptoms can distinguish organizational problems from technical ones and can identify which changes would address the root cause.
The second change is the deployment pipeline as a first-class engineering artifact. A team that does not have an automated deployment pipeline — or that has one that is unreliable and often bypassed — is working without the basic safety net that makes frequent delivery safe. Practitioners who have internalized continuous delivery treat the pipeline as critical infrastructure, prioritize pipeline health over feature delivery, and understand that a slow, unreliable pipeline is a bottleneck that affects everything downstream.
The third change is production empathy: the disposition to think about operational concerns during development. Code that is hard to observe, that produces unhelpful logs, that has no health endpoints, that fails catastrophically rather than gracefully is code that creates operational problems for whoever is responsible for it in production. The practitioner who has internalized operability writes code that is observable, that communicates its health, and that fails in diagnosable ways.
The fourth change is the recognition that frequent deployment is safer than infrequent deployment. This is counterintuitive — surely less change means less risk? — but the empirical evidence is the opposite. Large, infrequent deployments accumulate multiple changes, making failures impossible to isolate. Small, frequent deployments change one thing at a time, making failures easy to diagnose and revert. The risk is not in the act of deployment but in the size of the batch; reducing batch size through frequent deployment reduces risk.
Resources
Books and Texts
Kim, Humble, Debois, and Willis’s The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Your Technology Organization (IT Revolution Press, 2nd ed., 2021) is the most comprehensive practical implementation guide. The three ways of DevOps (Flow, Feedback, Continual Learning) organize the content; the specific practices are concrete and implementation-ready. The second edition added significant material on security integration (DevSecOps) and advanced deployment patterns.
Humble and Farley’s Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley, 2010) remains the foundational text on deployment pipelines despite being fifteen years old. Its treatment of pipeline design, test automation requirements, and the organizational changes required for continuous delivery is more thorough than any more recent treatment.
Kim, Behr, and Spafford’s The Phoenix Project (IT Revolution Press, 3rd ed., 2018) is the novel-format entry point — the fastest way to understand why DevOps matters and what it looks like to transform an organization. Its influence on non-technical readers who make organizational decisions has been enormous.
Forsgren, Humble, and Kim’s Accelerate (§7.3 reference) provides the empirical validation. Reading it alongside the DevOps Handbook demonstrates both that the practices work (Accelerate) and how to implement them (DevOps Handbook).
Morris’s Infrastructure as Code: Dynamic Systems for the Cloud Age (2nd ed., O’Reilly, 2021) is the most comprehensive treatment of the principles and practices of managing infrastructure through code. It covers the principles that apply across tools and the specific practices for making infrastructure code reliable and maintainable.
| Book | Role | Type |
|---|---|---|
| Kim, Humble, Debois & Willis, The DevOps Handbook (2nd ed.) | Comprehensive practical implementation | Practice |
| Humble & Farley, Continuous Delivery | Deployment pipeline foundational text | Depth |
| Kim, Behr & Spafford, The Phoenix Project (3rd ed.) | Novel-format accessible entry | Practice |
| Forsgren, Humble & Kim, Accelerate | Empirical foundation | Entry |
| Morris, Infrastructure as Code (2nd ed.) | IaC principles and practices | Practice |
Courses, Papers, and Current Sources
Debois’s original DevOpsDays talk recording and the associated blog posts document the origin of the term and the founding ideas. Reading the founding documents before the secondary literature provides useful perspective on what DevOps was meant to be, which differs in some ways from how it has been commercialized.
The State of DevOps reports (dora.dev, free, annual) provide the most current empirical data on DevOps adoption and its outcomes. Reading the most recent report and comparing to reports from previous years reveals which practices have sustained correlation with performance and how the measurement framework has evolved.
The Google SRE book (free at sre.google) and the Site Reliability Workbook (free at sre.google) provide the SRE model as the operational complement to DevOps development practices. Chapter 6 of the SRE book, on monitoring, is the most influential treatment of monitoring philosophy in the field.
| Resource | Platform | Type |
|---|---|---|
| State of DevOps reports (free) | dora.dev | Reference |
| Google SRE book and workbook (free) | sre.google | Depth |
Practice, Tools, and Current Sources
GitHub Actions (free for public repositories) and GitLab CI/CD (free tier) are the most accessible platforms for implementing deployment pipelines. Working through a complete pipeline — build, unit test, integration test, docker build and push, deployment to a staging environment — provides direct experience with pipeline design that reading cannot.
Terraform (free, terraform.io) with the Terraform Cloud free tier provides the most direct entry to infrastructure as code. Working through the official tutorials — provisioning cloud infrastructure, managing state, using modules — makes IaC concrete.
Feature flag tools — LaunchDarkly (commercial with free tier), Flagsmith (open source), or Unleash (open source) — implement feature flags. Working through the SDK integration for a feature flag and observing how a feature can be deployed but not enabled — then gradually enabled for increasing percentages of users — is the most direct way to understand how trunk-based development works in practice.
Argo CD (free, argoproj.io) implements GitOps for Kubernetes. Setting up Argo CD with a Git repository and observing how it detects drift between the desired state in Git and the actual cluster state, and how it reconciles automatically, makes GitOps operational.
| Resource | Platform | Type |
|---|---|---|
| GitHub Actions / GitLab CI (free) | GitHub / GitLab | Practice |
| Terraform (free tier) | terraform.io | Practice |
| Argo CD (free) | argoproj.io | Practice |
| Argo Rollouts (free) | argoproj.github.io | Practice |
| OpenTelemetry documentation (free) | opentelemetry.io | Practice |
| Backstage Software Catalog (free) | backstage.io | Practice |
| OpenFeature (free) | openfeature.dev | Practice |
| Flagsmith (open source) | flagsmith.com | Practice |
| Principles of Chaos Engineering (free) | principlesofchaos.org | Reference |
| Chaos Mesh / LitmusChaos (free) | CNCF / project docs | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| DevOps as a team name or role title | Naming a team “DevOps” and assigning it responsibility for CI/CD pipelines and infrastructure reproduces the wall of confusion in a new location: the DevOps team is the new silo between development and operations. The organizational change that DevOps requires is bringing operational concerns into development teams, not creating a new specialized team with the same separation dynamic. | DevOps is a cultural change that requires development teams to own what happens in production, not a team name. Platform engineering teams build the tooling that makes this possible; they are not themselves the DevOps team. If the development teams are not thinking about production observability, deployment safety, and operational burden, calling a team “DevOps” has not achieved DevOps. |
| Focusing on tooling before culture | Organizations that adopt Docker, Kubernetes, and Terraform without addressing the organizational separation between development and operations have improved their tooling without improving their process. The tools enable the practices; they do not implement them. | The DevOps Handbook’s three ways (Flow, Feedback, Continual Learning) are cultural and organizational, not technical. Address the incentive structures, team ownership models, and measurement systems before or alongside the tooling adoption. The DORA research identifies culture and organizational practices as predictors of performance alongside technical practices. |
| Treating continuous deployment as continuous delivery | Continuous deployment (every change that passes the pipeline is automatically deployed to production) and continuous delivery (the pipeline keeps software in a deployable state; deployment is a business decision) are different practices. Many organizations want continuous delivery but deploy to production only when a human decides to, for business reasons. Conflating them produces pressure to adopt continuous deployment before the organization is ready, or dismissal of continuous delivery as impractical. | Be precise about the distinction. Continuous delivery is achievable and valuable at most organizations. Continuous deployment is appropriate for some (high-volume consumer products where fast iteration is critical) and not for others (regulated industries, low-traffic internal tools). Choose deliberately based on the organizational context. |
| Manual stages in automated pipelines | A pipeline with a “manual QA approval” gate before production requires a human to verify each deployment. This bottleneck limits deployment frequency to whatever rate the QA team can review, introduces variability in review quality, and creates the batch accumulation that frequent deployment is supposed to prevent. Manual stages are often introduced because automated testing is insufficient to provide confidence; the solution is better automated testing, not a manual gate. | Identify the reason for each manual stage in the pipeline. If it is “we don’t trust the automated tests,” invest in automated tests that provide the required confidence. If it is “we need human judgment about release timing,” use a manual approval gate only on the final production deployment, not on intermediate stages. |
| Infrastructure as code without testing | Terraform or Ansible configurations that are committed without testing — that only get exercised when they are deployed to production — will fail in ways that surprise the team, often in irreversible ways. The same principle that applies to application code applies to infrastructure code: changes should be tested before production. | Test infrastructure code before deploying it. Terraform’s plan command shows what changes would be made; reviewing the plan before applying it is a minimal check. For more thorough validation: run infrastructure tests in a temporary environment (using Terraform workspaces or separate accounts), use tools like Terratest for automated infrastructure testing, and validate that the resulting infrastructure behaves as expected before teardown. |
中文
DevOps 不是一个角色、一个工具，也不是一种定义固定的方法论。它是一场文化与技术运动，回应的是一个具体问题：开发团队与运维团队之间的结构性对立。开发团队的目标是改变软件，运维团队的目标是保持软件稳定。当这些目标由不同群体承担，并且这些群体拥有不同激励、不同工具和不同知识时，结果就是交付缓慢、失败频繁，以及软件变更被当成紧急事件而非常规事件的组织。DevOps 试图解决这种对立：让开发团队拥有生产环境中发生事情的责任，同时让运维团队拥有使生产变更变得安全的手段。
构成 DevOps 的实践会汇聚到一个共同机制上：把一切可以自动化的东西自动化，使人类判断只保留给真正需要判断的决策，并使从 code 到 production 的路径足够可靠，以至于频繁小变更比低频大变更风险更低。一个需要 specialized team 每季度按四十步 manual checklist 执行的 deployment，必然会出错，团队也会尽量避免执行它，并且会积累大批量 change，使失败难以诊断。一个 fully automated、可由任何 developer 按需执行、并被自动监控的 deployment，则会被团队常规执行，产生小批量 change，并且失败更容易隔离和诊断。
Continuous delivery——让软件始终保持可发布状态，并能按需部署到 production——是 DevOps 组织目标的技术表达。它需要一个 automated deployment pipeline，使每次 code change 都经过一系列 environments 完成 build、test 和 deploy；它需要运行足够快、能在数分钟内提供反馈的 tests；也需要组织纪律，把 pipeline health 置于 feature delivery 之上。Continuous deployment 是更严格的实践：每个 change 只要通过全部 pipeline stages，就会在没有人工批准的情况下自动部署到 production。
前置知识：测试（§7.2）——automated tests 是使快速部署变安全的基础。软件过程（§7.3）——DevOps 是一种具有组织含义的过程实践。容器编排（§4.9）——现代部署基础设施。可观测性与可靠性工程（§4.10）——production monitoring 是 DevOps 实践的中心。
从 Wall of Confusion 到 Platform Engineering
DevOps 所处理的问题，是由一种当初看起来直观合理的组织结构制造出来的。
在 1990 年代和 2000 年代早期，组织通常会把软件开发与运维分开。开发团队写软件；运维团队部署并维护软件。这种分离反映了真实关切：developers 对 production systems 做 ad hoc changes 可能造成 outages；一个具有受控访问权限的 specialized operations team 可以维持稳定性。这种分离也反映了组织逻辑：写代码的技能与管理 servers 的技能不同，按技能分组是自然的。
问题是结构性的。开发团队按 shipping features 衡量；运维团队按 uptime 衡量。Feature shipping 需要 change；uptime 需要 stability。激励直接冲突，而这种冲突表现为一堵墙：development 把软件扔过墙交给 operations，operations 在它部署失败时再扔回来，双方互相责怪。Deployments 变成低频、计划性事件——每季度一次甚至更少——因为每次 deployment 都要求多个优先级冲突的团队协调、进行大量 manual verification，并安排 maintenance windows。低频 deployments 会积累大批 change，使失败不可能被隔离和诊断。这堵后来在 DevOps community 中被称为 wall of confusion 的墙，使 development 和 operations 都更难达成自己的目标。
Patrick Debois 是一位 Belgian consultant，他一直在研究弥合 development 与 operations 之间差距的方法。2009 年，他在 Ghent 组织第一届 DevOpsDays conference 时使用了 “DevOps” 这个词。这个名字保留下来。这个运动迅速增长，因为它处理了许多组织都在自己身上识别出的真实问题。Agile movement 改善了开发实践，但没有处理 development 与 operations 之间的 handoff；DevOps 补完了这个回路。
Gene Kim、Kevin Behr 和 George Spafford 的 The Phoenix Project（2013）通过小说形式把 DevOps 带给了更广泛受众：它讲述一个陷入危机的 IT organization，跟随主人公 Bill Palmer 发现并实施 DevOps 的 Three Ways。Three Ways——Flow（快速、顺畅地交付工作）、Feedback（快速看到并解决问题）和 Continual Learning（安全实验，并从成功和失败中学习）——提供了理解具体 DevOps practices 为什么有效的框架。The Phoenix Project 可以说比任何技术书都更有影响力，因为它触达了那些永远不会阅读技术实施指南的 managers 和 executives。
Infrastructure-as-code movement 与 DevOps movement 并行发展，并且对它至关重要。如果基础设施是通过在 servers 上运行 commands 管理的，就没有 audit trail、没有 repeatability，也没有在失败后不依赖人类记忆恢复的方式。如果基础设施被定义为 code——无论是在 Puppet（2005）、Chef（2009）、Ansible（2012）这样的 configuration management tools 中，还是在 Terraform（2014）这样的 declarative provisioning tools 中——它就变得可测试、可版本化、可复现。Provision development environment 的同一份 code，也 provision production environment；环境之间的差异会显示在 code diffs 中，而不是表现为神秘的行为差异。
Docker 于 2013 年推出，通过让 containers——轻量、可移植、可复现的执行环境——对应用部署变得实用，改变了运维模型。Docker 之前，deployments 很复杂，因为同一个 application 会由于 installed packages、configuration files 和 library versions 的差异，在不同 servers 上表现不同。Docker container 会把 application 及其全部 dependencies 打包成一个 single artifact，并且无论部署到哪里都以相同方式运行。这消除了由 environment inconsistency 造成的一类 bugs，并使 “build once, run anywhere” 的分离变得可实现。再结合用于 orchestration 的 Kubernetes（§4.9），Docker containers 成为 cloud-native applications 的主导 deployment artifact。
DORA research program（§7.3）提供了经验证据，使 DevOps movement 获得超越叙事和直觉的可信度。它发现 elite performers 可以按需部署，同时保持低 change failure rates；而且这种组合预测的是 organizational performance，不只是 delivery performance。这证明 DevOps practices 不只是 developer preferences，而是 business differentiators。它识别出的预测 elite performance 的技术实践——trunk-based development、continuous integration、deployment automation、monitoring and observability、test automation、database change management——成为 DevOps 在技术上要求什么的经典列表。
Google 的 Site Reliability Engineering model（SRE，§4.10）处理的是相关但不同的问题：如何让大型复杂系统在规模化下可靠运行。SRE 通过 Service Level Objectives 定义 reliability targets，使用 error budgets 平衡 reliability 和 velocity，把 toil（可以自动化的 manual operational work）视为需要消除的敌人，并要求 SRE engineers 把相当一部分时间用于写 code 来自动化 operations，而不是手动执行 operations。SRE 和 DevOps 是互补的：DevOps 处理 development 与 operations 之间的组织结构；SRE 处理高可靠系统中 operations 内部的实践。
Platform engineering 在 2010 年代后期和 2020 年代早期出现，是 DevOps movement 的成熟形态。随着组织采用 DevOps practices，它们发现，要求每个 development team 独立管理自己的 deployment pipelines、observability infrastructure 和 cloud provisioning，会产生不一致和重复劳动。Platform engineering teams 构建 internal developer platforms——也就是 tooling、APIs 和 abstractions，使 development teams 更容易做正确的事情。Internal Developer Platform 为常见任务提供 golden paths（部署新 service、设置 observability、管理 secrets），这些路径实现 best practices，而不要求每个团队独立发现这些实践。Team Topologies framework（§7.3）把 platform engineering team 归类为 platform team，向 stream-aligned development teams 提供 X-as-a-Service。
Deployment Pipeline、Infrastructure as Code 与 Observability
Deployment Pipeline 作为组织骨架
Deployment pipeline 是每次 code change 从 developer commit 到 production deployment 所经过的一系列自动化 stages。其结构反映了一个原则：反馈应尽可能早、尽可能便宜地到来。更快、更便宜的 stages 先运行；更慢、更昂贵的 stages 后运行。
一个典型 pipeline 有几个 stages。Commit stage 会在每次 code push 后立即运行，编译 code，运行 unit tests，执行 static analysis，并产生 deployable artifact。这个 stage 应在五分钟或更短时间内完成；如果它太慢，developers 就无法得到快速反馈，并会以更大 batches 工作。Acceptance stage 会在 production-like environment 中针对已部署 artifact 运行 integration tests 和 automated acceptance tests；它提供系统整体按预期工作所需的 verification。Performance tests 和 security scans 可以并行运行，或放在后续 stages 中。Production deployment stage 会把软件部署到 production——要么在通过前面全部 stages 后自动部署（continuous deployment），要么带有人类 approval gate（continuous delivery）。
Pipeline 要正常工作，需要若干支撑实践。Artifact promotion——在所有 pipeline stages 中使用同一个 artifact（同一个 container image 或 compiled binary），而不是每个 stage 重新 build——确保被测试的东西就是被部署的东西。Environment parity——使 development、staging 和 production environments 尽可能相似——确保 staging tests 能捕捉 production 中会出现的问题。Pipeline health monitoring——把 failing pipeline stage 当作 production incident 一样紧急处理——确保 pipeline 提供可靠反馈，而不是变成团队学会忽视的噪音源。
Feature flags（也叫 feature toggles）允许 code 在 feature 对 users 启用之前就部署到 production。Flag 把 deployment 与 release 分离：code 已在 production，但 feature 尚不可见。这使 trunk-based development 成为可能，即使某些 features 需要数周开发，也可以让所有 changes merge 到始终可部署的 main 上。它也支持 gradual rollout——先为 1% users 启用 feature，再到 10%，最后到 100%——从而减少 bugs 的 blast radius。它还支持 A/B testing——不同 users 获得 feature 的不同版本——从而支持经验性优化。Feature flags 需要管理：不再需要的 flags 应被移除，而大量 active flags 会在 code 中制造 combinatorial complexity。
Blue-green deployment 和 canary releases 是降低 production deployments 风险的策略。在 blue-green deployment 中，两个 production environments（blue 和 green）同时存在；deployment 时 traffic 从一个切换到另一个，rollback 则通过把 traffic 切回原环境立即完成。在 canary release 中，一小部分 production traffic（例如 1%）被路由到新版本，其余继续使用旧版本；系统监控 metrics，只有新版本表现符合预期时才提高比例。两种策略都允许在问题影响所有 users 之前，早期发现 production problems。
Infrastructure as Code 与 Configuration Management
Infrastructure as code（IaC）把基础设施的 provisioning 和 configuration——servers、networks、databases、load balancers——当作 software，用与 application code 相同的工具（version control、code review、automated testing）管理。其收益包括 repeatability（应用同一配置产生同一结果）、auditability（每次 change 都是带 author 和 message 的 commit），以及 disaster recovery（基础设施可以从 code 复现，而不是从记忆中恢复）。
Terraform（HashiCorp，2014）是 declarative infrastructure provisioning 的主导工具。Terraform configuration 描述 cloud infrastructure 的 desired state——哪些 virtual machines、哪些 networking、哪些 managed services——然后 Terraform 计算需要做出哪些 changes，才能使 actual state 与 desired state 对齐。Plan/apply workflow——计算 changes、review 它们、apply 它们——使人在 changes 发生前就能看到将发生什么。Terraform state——Terraform 认为当前已经 provisioned 的内容记录——是必须谨慎管理的关键 artifact；state corruption 或 inconsistency 会产生不可预测行为。
Ansible、Puppet 和 Chef 处理 configuration management——也就是对 servers 上安装的软件及其配置进行持续管理。在 cloud-native environments 中，这些工具不再那么中心，因为 immutable infrastructure（servers 被替换，而不是被更新）已经成为 deployment model；但对于早于 containerization 的系统，或无法 containerize 的系统，它们仍然重要。
GitOps 是从 Kubernetes ecosystem 中出现的实践，它使用 Git repository 作为 cluster configuration 的 source of truth。GitOps controller（Argo CD、Flux）会持续比较 repository 中描述的 desired state 与 cluster 的 actual state，并应用 changes，使 cluster 对齐。基础设施的每一次 change 都是一个 pull request，带有 review、approval 和 audit trail。Rollback 就是一个 revert commit。GitOps 是 Kubernetes deployments 中最干净的 infrastructure as code 实现。
测量与监控 Production
Observability——通过外部输出理解 production system 内部状态的能力——是让频繁部署变安全的基础。如果团队看不到自己的软件在 production 中正在做什么，就无法知道某次 deployment 是否成功、某个行为变化是否是 regression，或 users 是否正在经历问题。没有 observability，频繁部署是鲁莽；有了 observability，频繁部署才是风险管理。
DORA research 把 monitoring and observability 识别为最强预测软件交付表现的技术实践之一。拥有良好 observability 的团队能更快检测和诊断问题，使较小 error budgets 被花在学习上，而不是被漫长 outages 消耗。具体实践——structured logging、具有 meaningful cardinality 的 metrics、distributed tracing，以及越来越重要的 continuous profiling——见 §4.10。
Deployment frequency measurement 是 deployment pipeline 本身最简单的 observability signal。能够追踪自己多频繁部署到 production、并看到趋势随时间变化的团队，拥有关于 process health 的最基本 feedback loop。一个每月部署一次、想要每天部署的团队，有一个清晰、可测量的目标；一个不追踪 deployment frequency 的团队，则无法知道自己是否正在改进。
学习这一部分会改变什么
DevOps 会改变实践者理解软件开发与生产运维关系的方式。
第一个变化，是能够识别 wall of confusion 的成本。那些 development 和 operations 是分离团队、且拥有分离激励的组织，会产生典型症状：低频 deployments、高 change failure rates、长 recovery times。理解这些症状结构性来源的实践者，可以区分组织问题和技术问题，并识别哪些变化能处理 root cause。
第二个变化，是把 deployment pipeline 视为一等工程产物。一个没有 automated deployment pipeline 的团队——或拥有一个不可靠、经常被绕过的 pipeline 的团队——是在缺少基本 safety net 的情况下工作，而这个 safety net 正是让 frequent delivery 变安全的基础。内化 continuous delivery 的实践者，会把 pipeline 当作 critical infrastructure，把 pipeline health 置于 feature delivery 之上，并理解 slow、unreliable pipeline 是影响下游一切工作的瓶颈。
第三个变化，是 production empathy：在开发过程中思考运维关切的倾向。难以观察、产生无用 logs、没有 health endpoints、灾难性失败而不是 graceful failure 的 code，会为负责其 production 的人制造运维问题。内化 operability 的实践者，会写出 observable 的 code，能沟通自身 health，并以可诊断方式失败。
第四个变化，是认识到频繁部署比低频部署更安全。这一点反直觉——难道更少 change 不意味着更少 risk 吗？——但经验证据正相反。大型、低频 deployments 会积累多个 changes，使失败不可能被隔离。小型、频繁 deployments 一次只改变一件事，使失败易于诊断和 revert。风险不在 deployment 这个动作本身，而在 batch 的大小；通过频繁部署减少 batch size，就会降低风险。
资源
书籍与文本
Kim、Humble、Debois 和 Willis 的 The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Your Technology Organization（IT Revolution Press，第 2 版，2021）是最综合的实践实施指南。DevOps 的 three ways（Flow、Feedback、Continual Learning）组织全书内容；具体 practices 清楚且可实施。第二版增加了大量关于 security integration（DevSecOps）和 advanced deployment patterns 的内容。
Humble 和 Farley 的 Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation（Addison-Wesley，2010）虽然已经出版十五年，但仍然是 deployment pipelines 的奠基文本。它对 pipeline design、test automation requirements，以及 continuous delivery 所需组织变化的处理，比任何较新的文本都更彻底。
Kim、Behr 和 Spafford 的 The Phoenix Project（IT Revolution Press，第 3 版，2018）是小说形式入口——这是理解 DevOps 为什么重要，以及组织转型是什么样子的最快方式。它对那些做组织决策的非技术读者产生了巨大影响。
Forsgren、Humble 和 Kim 的 Accelerate（§7.3 参考）提供经验证据。把它与 DevOps Handbook 一起阅读，可以同时看到这些 practices 为什么有效（Accelerate）以及如何实施它们（DevOps Handbook）。
Morris 的 Infrastructure as Code: Dynamic Systems for the Cloud Age（第 2 版，O’Reilly，2021）是关于通过 code 管理 infrastructure 的原则与实践的最综合处理。它覆盖跨工具适用的原则，以及使 infrastructure code 可靠且可维护的具体实践。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Kim, Humble, Debois & Willis, The DevOps Handbook（第 2 版） | 综合实践实施 | 实践 |
| Humble & Farley, Continuous Delivery | Deployment pipeline 奠基文本 | 深入 |
| Kim, Behr & Spafford, The Phoenix Project（第 3 版） | 小说形式易读入口 | 实践 |
| Forsgren, Humble & Kim, Accelerate | 经验基础 | 入门 |
| Morris, Infrastructure as Code（第 2 版） | IaC 原则与实践 | 实践 |
课程、论文与当前资料
Debois 的 original DevOpsDays talk recording 及相关 blog posts 记录了这个术语的起源和奠基思想。在阅读二手文献前先读奠基文件，有助于理解 DevOps 原本意图是什么，而这与它被商业化后的样子在某些方面不同。
State of DevOps reports（dora.dev，免费，每年发布）提供关于 DevOps adoption 及其 outcomes 的最新经验数据。阅读最新 report，并与前几年 reports 比较，可以看到哪些 practices 与 performance 的相关性持续存在，以及 measurement framework 如何演化。
Google SRE book（可在 sre.google 免费获取）和 Site Reliability Workbook（可在 sre.google 免费获取）提供 SRE model，作为 DevOps development practices 的运维补充。SRE book 第 6 章关于 monitoring 的内容，是这个领域最有影响力的 monitoring philosophy 处理。
| 资源 | 平台 | 类型 |
|---|---|---|
| State of DevOps reports（免费） | dora.dev | 参考 |
| Google SRE book and workbook（免费） | sre.google | 深入 |
实践、工具与当前资料
GitHub Actions（公共 repositories 免费）和 GitLab CI/CD（有免费层）是实现 deployment pipelines 最易获得的平台。完成一条完整 pipeline——build、unit test、integration test、docker build and push、deployment to a staging environment——可以获得阅读无法提供的 pipeline design 直接经验。
Terraform（免费，terraform.io）配合 Terraform Cloud 免费层，是进入 infrastructure as code 最直接的路径。完成官方 tutorials——provisioning cloud infrastructure、managing state、using modules——会使 IaC 变得具体。
Feature flag tools——LaunchDarkly（商业，有免费层）、Flagsmith（开源）或 Unleash（开源）——实现 feature flags。完成某个 feature flag 的 SDK integration，观察一个 feature 如何被 deployed 但尚未 enabled，然后如何逐步对越来越高比例的 users 启用，是理解 trunk-based development 实践运作方式的最直接方法。
Argo CD（免费，argoproj.io）为 Kubernetes 实现 GitOps。用 Git repository 设置 Argo CD，观察它如何检测 Git 中 desired state 与 actual cluster state 之间的 drift，以及如何自动 reconcile，会使 GitOps 的运维形式具体化。
| 资源 | 平台 | 类型 |
|---|---|---|
| GitHub Actions / GitLab CI（免费） | GitHub / GitLab | 实践 |
| Terraform（免费层） | terraform.io | 实践 |
| Argo CD（免费） | argoproj.io | 实践 |
| Argo Rollouts（免费） | argoproj.github.io | 实践 |
| OpenTelemetry documentation（免费） | opentelemetry.io | 实践 |
| Backstage Software Catalog（免费） | backstage.io | 实践 |
| OpenFeature（免费） | openfeature.dev | 实践 |
| Flagsmith（开源） | flagsmith.com | 实践 |
| Principles of Chaos Engineering（免费） | principlesofchaos.org | 参考 |
| Chaos Mesh / LitmusChaos（免费） | CNCF / project docs | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 把 DevOps 当成 team name 或 role title | 把一个团队命名为 “DevOps”，并让它负责 CI/CD pipelines 和 infrastructure，会在新位置重建 wall of confusion：DevOps team 成为 development 与 operations 之间的新 silo。DevOps 要求的组织变化，是把 operational concerns 带入 development teams，而不是创建一个新的 specialized team，重复同样的分离动态。 | DevOps 是一种文化变化，要求 development teams 拥有 production 中发生事情的责任，而不是一个团队名称。Platform engineering teams 构建使这一点成为可能的 tooling；它们本身不是 DevOps team。如果 development teams 没有思考 production observability、deployment safety 和 operational burden，那么把某个团队叫做 “DevOps” 并没有实现 DevOps。 |
| 先关注 tooling，而不是 culture | 组织采用 Docker、Kubernetes 和 Terraform，却不处理 development 与 operations 之间的组织分离，只是改善了 tooling，而没有改善 process。工具使 practices 成为可能；它们不会自动实施这些 practices。 | DevOps Handbook 的 three ways（Flow、Feedback、Continual Learning）是文化和组织性的，不是技术性的。在采用 tooling 之前或同时，处理 incentive structures、team ownership models 和 measurement systems。DORA research 把 culture 和 organizational practices 与 technical practices 一起识别为 performance predictors。 |
| 把 continuous deployment 当成 continuous delivery | Continuous deployment（每个通过 pipeline 的 change 都自动部署到 production）和 continuous delivery（pipeline 使软件保持可部署状态；是否部署是 business decision）是不同实践。许多组织想要 continuous delivery，但出于业务原因，只在人工决定时才部署到 production。混淆二者，会导致组织在尚未准备好时被迫采用 continuous deployment，或把 continuous delivery 误判为不现实。 | 精确区分二者。Continuous delivery 在多数组织中都可实现且有价值。Continuous deployment 适合某些组织（快速迭代关键的高流量 consumer products），不适合另一些组织（regulated industries、low-traffic internal tools）。应根据组织语境有意识地选择。 |
| Automated pipelines 中的 manual stages | 一个在 production 前有 “manual QA approval” gate 的 pipeline，要求人类验证每次 deployment。这个 bottleneck 会把 deployment frequency 限制在 QA team 能 review 的速度上，引入 review quality 的变异性，并制造 frequent deployment 本来要防止的 batch accumulation。Manual stages 通常是因为 automated testing 不足以提供信心而被引入；解决方案应是更好的 automated testing，而不是 manual gate。 | 识别 pipeline 中每个 manual stage 存在的理由。如果理由是“我们不信任 automated tests”，就投资能提供所需信心的 automated tests。如果理由是“我们需要人类判断 release timing”，则只在最终 production deployment 上使用 manual approval gate，而不是在中间 stages 上使用。 |
| Infrastructure as code 没有 testing | 没有经过 testing 的 Terraform 或 Ansible configurations——也就是只有部署到 production 时才真正执行的配置——会以让团队意外的方式失败，而且常常以不可逆方式失败。适用于 application code 的原则同样适用于 infrastructure code：changes 应在 production 前被测试。 | 在部署前测试 infrastructure code。Terraform 的 plan command 会显示将做出哪些 changes；在 apply 前 review plan 是最低限度检查。更彻底的 validation 包括：在临时环境中运行 infrastructure tests（使用 Terraform workspaces 或独立 accounts），使用 Terratest 等工具进行 automated infrastructure testing，并在 teardown 前验证生成的 infrastructure 按预期行为运行。 |