English
Infrastructure used to be managed through instructions: run this script on this machine, configure this service this way, install this package in this order. When something went wrong, an operator diagnosed the problem and issued more instructions to correct it. This model is understandable at small scale. At scale — hundreds of services, thousands of machines, continuous deployment many times per day — it becomes untenable. The mental model of what is running where cannot be held in any number of human heads. The imperative scripts that put software into production produce different outcomes depending on the prior state of the machine. Failures require manual intervention that cannot keep pace with the failure rate.
Container orchestration is the discipline’s answer to this scaling problem. The shift at its center sounds simple but runs deep: infrastructure should be described by what it should be, not instructed by the steps to make it so. An orchestration system compares the described desired state to the actual state of the cluster and continuously takes action to close the gap — starting workloads that should be running but are not, terminating workloads that should not be running but are, replacing failed instances with healthy ones. Human operators manage the description; the system manages the reconciliation. This is not merely a convenience; it is a fundamentally different operational model in which correctness is defined by desired state, and convergence to that state is the system’s responsibility rather than the operator’s.
A container is a running instance of a container image — a portable, self-contained package of application code together with its dependencies, runtime libraries, and configuration. Containers use Linux kernel features (namespaces for isolation, cgroups for resource limits) to run multiple isolated processes on a shared operating system kernel, with stronger isolation guarantees than processes but less overhead than virtual machines. A container image captures all of this in an immutable, versionable artifact that runs identically in development, testing, and production. The immutability is architectural: the same image that was tested is the image that runs in production, eliminating the class of “works on my machine” failures that plagued earlier deployment models.
Prerequisites: Operating systems (§4.2) — Linux namespaces, cgroups, and process isolation are the kernel primitives on which containers rest. Computer networks (§4.3) — service discovery, load balancing, and network policy are core orchestration concerns. Distributed systems (§4.6) — orchestration systems are distributed systems; etcd, the Kubernetes configuration store, is a Raft-based distributed key-value store.
From Borg to Kubernetes and the Cloud-Native Ecosystem
Google’s container orchestration history predates Kubernetes by a decade. Borg, Google’s internal cluster management system, was developed starting around 2003 and had been managing Google’s production workloads at scale for years before Kubernetes was announced. Borg introduced the ideas that Kubernetes would later make public: declarative job specifications, a centralized control plane that reconciled desired state against actual state, resource-request-and-limit scheduling across shared physical machines, and the notion of a container as the unit of deployment. The engineers who built Borg brought its lessons to Kubernetes — including the lesson that the earliest design choices constrain the system for decades.
Docker, introduced by Solomon Hykes at PyCon 2013, was not a scheduler or orchestration system; it was a tool that made Linux containers usable. Before Docker, Linux containers (LXC) required deep system administration knowledge to use. Docker provided a simple command-line interface, a standard container image format (layers over a union filesystem), and a registry (Docker Hub) for sharing images. Within months of its introduction, Docker had made container-based deployment accessible to practitioners who would never have configured LXC directly. The adoption was rapid and consequential: container images became the standard unit of packaging, and the question of orchestrating containers at scale became urgent.
Google announced Kubernetes in June 2014, open-sourcing it as a redesign of Borg that incorporated lessons from both Borg and from Omega, a more flexible successor. Where Borg had proprietary APIs and was tightly coupled to Google’s internal infrastructure, Kubernetes was designed to run on commodity cloud infrastructure with open, extensible APIs. The three authors of the original Kubernetes announcement paper — Brendan Burns, Brian Grant, and David Oppenheimer — framed it explicitly as a vehicle for sharing what Google had learned about running containers at scale. Kubernetes was contributed to the Cloud Native Computing Foundation (CNCF) in 2016, and the CNCF became the organizational center for the cloud-native ecosystem.
The CNCF model — a vendor-neutral foundation governing open-source infrastructure projects — enabled the ecosystem to develop without being controlled by any single vendor. Projects for container networking (Calico, Cilium, Flannel), service mesh (Istio, Linkerd), certificate management (cert-manager), secrets management (HashiCorp Vault), continuous delivery (Argo CD, Flux), monitoring (Prometheus), distributed tracing (Jaeger, Zipkin), and dozens of other concerns were governed under a common umbrella. The CNCF Landscape, mapping over a thousand projects across the cloud-native space, reflects both the ecosystem’s richness and the navigation challenge it poses for practitioners: choosing among functionally similar tools requires understanding design tradeoffs that are not evident from their surface descriptions.
The service mesh, which emerged around 2017, addressed a specific orchestration gap: managing service-to-service communication in a cluster with many independent services. A service mesh injects a proxy sidecar alongside every service instance; the sidecar handles TLS, load balancing, circuit breaking, and telemetry, allowing these concerns to be managed uniformly across services without modifying service code. Istio, built on the Envoy proxy, became the dominant service mesh; Linkerd offered a lighter-weight alternative. The sidecar model imposed overhead — a proxy alongside every pod — that became significant at scale. The emergence of eBPF-based alternatives (Cilium, which implements service-mesh functionality in the kernel rather than in sidecars) and sidecarless mesh models (Istio Ambient mode) is restructuring the service mesh space, but the underlying problem — managing east-west service communication in a polyglot, multi-service cluster — remains a central orchestration concern.
GitOps emerged as the operational model that completed declarative infrastructure management. If infrastructure is described in version-controlled files, then the state of the infrastructure at any point can be derived from the state of the repository at that point. Argo CD and Flux are GitOps controllers that continuously synchronize the cluster’s actual state with the desired state declared in a Git repository, automatically applying changes when the repository is updated and detecting and correcting drift between the cluster’s actual state and the declared state. GitOps changes the audit trail: every change to production infrastructure is a reviewed, approved commit in the repository, with a history that extends back to the beginning of the project. The operational model it enables — operators review pull requests rather than executing scripts — is a significant improvement in both safety and transparency over imperative deployment procedures.
The most recent structural change is the convergence of cloud-native infrastructure with AI/ML workloads. GPU scheduling requires orchestration primitives that were not designed into Kubernetes: GPUs are not interchangeable resources, fractional GPU sharing has correctness implications, and training jobs that require specific interconnect topologies (NVLink, InfiniBand) need placement logic that generic bin-packing schedulers do not provide. KubeRay (distributed Ray on Kubernetes), the Kubeflow platform, and various GPU-aware scheduler extensions (NVIDIA GPU Operator, custom scheduler plugins) represent the engineering response. The inference serving pattern — routing requests to service instances with the right model loaded, with appropriate batching and memory management — stresses the Kubernetes service model in ways that are still being worked out. AI workloads are producing a second wave of orchestration innovation comparable to the original wave that produced Kubernetes.
Reconciliation, Scheduling, and the Developer-Operator Boundary
Declarative State and the Reconciliation Loop
Every Kubernetes resource — Pod, Deployment, Service, Ingress, ConfigMap, PersistentVolumeClaim — is a declaration of desired state stored in etcd, the cluster’s distributed configuration store. Controllers watch these resources, detect discrepancy between desired and actual state, and take action to close the gap. A Deployment controller notices that the number of running Pods does not match the declared replica count and creates or terminates Pods accordingly. A scheduler watches for unscheduled Pods and places them on nodes that satisfy their constraints. A kubelet watches for Pods assigned to its node and starts or stops containers to match the declaration.
The reconciliation loop is the architectural primitive of cloud-native infrastructure. Understanding it is understanding Kubernetes. Controllers are not scripts that run once; they run continuously, re-evaluating the discrepancy between desired and actual state after every relevant event. This means the system self-corrects: if a Pod is deleted by a node failure, the Deployment controller detects the discrepancy and creates a replacement. If a Service’s endpoints change because Pods are restarted, the Endpoints controller updates the service routing accordingly. The system converges to desired state without manual intervention in most failure scenarios.
The operator pattern extends reconciliation to application-specific operational concerns. A Kubernetes Operator is a custom controller managing a custom resource that extends the Kubernetes API with application-specific semantics. An Elasticsearch Operator knows how to safely scale an Elasticsearch cluster (waiting for shard rebalancing before removing nodes), how to perform rolling upgrades (updating one node at a time, verifying health before proceeding), and how to take coordinated backups. A database Operator manages failover, replica promotion, and backup scheduling. Operators encode operational knowledge that would otherwise require human expertise, making it automatable and repeatable. The operator pattern is why the CNCF ecosystem includes operators for hundreds of stateful applications; it is also why running stateful workloads on Kubernetes is more tractable than early practitioners expected.
Scheduling: Placement Decisions and Their Consequences
The Kubernetes scheduler’s job is to assign each unscheduled Pod to a node that satisfies its constraints. The constraints come from many sources: resource requests (CPU and memory the Pod requires), node affinity and anti-affinity rules (node labels the Pod prefers or requires, or other Pods it should be placed near or away from), topology spread constraints (distributing Pods across zones or nodes for resilience), and pod disruption budgets (limiting the number of Pods that can be unavailable simultaneously during voluntary disruptions).
Resource requests and limits are the most consequential scheduling parameters for most workloads. A Pod’s resource request declares what it needs; the scheduler places it on a node with sufficient available capacity. A Pod’s resource limit declares the maximum it can use; the kubelet enforces this by throttling CPU and OOM-killing processes that exceed memory limits. The difference between request and limit is slack — headroom that allows Pods to burst above their requested resources when the node has spare capacity. Getting resource requests right is one of the practical challenges of running Kubernetes: requests too low lead to scheduling onto overcommitted nodes that degrade under load; requests too high prevent efficient bin-packing and waste cluster resources.
Workload isolation in multi-tenant clusters requires understanding that Kubernetes namespace boundaries do not provide strong security isolation — Pods in different namespaces on the same node share the kernel and can potentially interfere. Strong isolation requires either network policies (restricting which Pods can communicate), Pod Security Standards (enforcing security contexts that limit privilege escalation), or physical separation (separate node pools or separate clusters). The Kubernetes security model is a layered concern, not a single control.
The Developer-Operator Interface
Cloud-native infrastructure interposes abstractions between application developers and the operational substrate. Developers declare their application’s needs (resource requirements, configuration, health endpoints, scaling behavior); the platform provides the execution environment. This abstraction allows developers to deploy applications without deep infrastructure knowledge, and it allows operators to evolve the infrastructure without breaking application declarations.
The abstraction design question — which concerns to expose to developers, which to hide, how to allow operational teams to configure defaults — is the central challenge of platform engineering. A platform that exposes too much infrastructure complexity forces developers to understand details irrelevant to their work. A platform that hides too much prevents developers from expressing requirements the infrastructure can satisfy. Platform engineering, as a discipline, applies deliberate product thinking to the design of the internal developer platform: defining user personas, understanding developer workflows, building feedback loops, and evolving the platform based on developer experience.
Helm, the Kubernetes package manager, addresses the YAML complexity problem by introducing templating. A Helm chart packages related Kubernetes resources with configurable parameters, allowing a single chart to deploy a Redis instance with different configurations in development and production. Kustomize takes a different approach — overlay-based configuration without templating — that preserves YAML’s simplicity while allowing environment-specific customization without duplicating base configuration. Both address the practical problem that raw YAML at scale becomes unmaintainable; the choice between them reflects preferences about complexity management.
What Studying This Changes
Container orchestration changes how practitioners think about deployment and operational reliability.
The first change is the ability to reason about infrastructure operationally, not just deploy it. This means diagnosing why a Pod is pending (insufficient node resources, scheduler constraint violations, node affinity mismatches), why a Service is not routing (endpoint selection failures, network policy blocks, readiness probe failures), why a Deployment rollout is stalled (maxUnavailable and maxSurge settings, readiness gates, pod disruption budget violations). This diagnostic capacity requires understanding what the orchestration system is doing internally, not merely how to express desired state to it.
The second change is judgment about when cloud-native infrastructure is appropriate. The discipline provides genuine capability for workloads that require horizontal scalability, continuous deployment, multi-service coordination, and geographic distribution. It imposes genuine overhead — operational complexity, team expertise requirements, infrastructure costs — on workloads that do not have those requirements. Practitioners who have internalized the discipline can assess honestly which characteristics justify the investment, rather than defaulting to Kubernetes because it is the current standard.
The third change is application design with the deployment substrate in mind. Cloud-native applications are designed for ephemerality: containers may be moved or restarted at any time, so application state must be externalized (to databases, object storage, or distributed caches) rather than held locally. They expose health endpoints (liveness and readiness probes) that the orchestrator uses to route traffic and restart unhealthy instances. They emit structured telemetry suitable for distributed observation. These requirements follow from the orchestration system’s design, and applications that violate them produce failure modes the orchestration system cannot automatically recover from.
Resources
Books and Texts
Hightower, Burns, and Beda’s Kubernetes: Up and Running (3rd ed., O’Reilly, 2022) is the canonical introduction. Written by three of the engineers most responsible for Kubernetes’s early development, it presents the system as its designers intended — covering Pods, Services, Deployments, and the major workload types with explanations of what each concept is and why it exists. The building-up approach makes the system comprehensible rather than a collection of YAML schemas to memorize.
Burns’s Designing Distributed Systems (O’Reilly, 2018) covers the distributed-systems patterns that cloud-native infrastructure makes practical — sidecars, ambassadors, leader election, work queues, scatter-gather — with a practitioner orientation. It provides the vocabulary of cloud-native application design. Ibryam and Huss’s Kubernetes Patterns (2nd ed., O’Reilly, 2023) covers the recurring design patterns of cloud-native applications specifically — how to structure health probes, configuration injection, leader election, and other concerns within Kubernetes constraints. Davis’s Cloud Native Patterns addresses application architecture more broadly.
For production operation, Production Kubernetes (Vyas, Love, Heller, Wood, O’Reilly, 2021) covers security hardening, multi-tenancy, and operational tooling for organizations running their own clusters. For infrastructure as code, Brikman’s Terraform: Up and Running (3rd ed., O’Reilly, 2022) is the standard Terraform introduction.
| Book | Role | Type |
|---|---|---|
| Hightower, Burns & Beda, Kubernetes: Up and Running (3rd ed.) | Canonical introduction | Entry |
| Burns, Designing Distributed Systems | Cloud-native architectural patterns | Entry |
| Ibryam & Huss, Kubernetes Patterns (2nd ed.) | Design patterns reference | Reference |
| Vyas et al., Production Kubernetes | Production operation depth | Depth |
| Davis, Cloud Native Patterns | Application architecture for cloud-native | Depth |
| Brikman, Terraform: Up and Running (3rd ed.) | Infrastructure as code | Entry |
| Skelton & Pais, Team Topologies | Platform engineering organizational foundation | Depth |
Courses, Papers, and Current Sources
Hightower’s Kubernetes the Hard Way (free online at github.com/kelseyhightower/kubernetes-the-hard-way) guides the learner through assembling a Kubernetes cluster from constituent components — the API server, controllers, scheduler, kubelets, networking, storage. This exercise is demanding and produces understanding that books cannot: engineers who have assembled a cluster know what managed services hide and can reason about it when something goes wrong. It is the essential depth exercise, best done after reading Kubernetes: Up and Running.
The foundational papers are Verma et al.’s “Large-scale cluster management at Google with Borg” (EuroSys 2015, free) and Burns et al.’s “Borg, Omega, and Kubernetes” (ACM Queue 2016, free). Together they explain where Kubernetes came from and which design lessons its creators carried from production experience at Google. Reading them makes the system more legible — many Kubernetes design choices that appear arbitrary are responses to Borg lessons that the papers document.
KubeCon and CloudNativeCon talks (free on YouTube) represent the contemporary practitioner state of the field. The CNCF YouTube channel archives talks from every annual conference; filtering by topic (security, networking, platform engineering, AI/ML workloads) produces practical coverage of current concerns.
| Course/Resource | Platform | Type |
|---|---|---|
| Verma et al., “Large-scale cluster management at Google with Borg” (free) | Borg architecture paper | Depth |
| Burns et al., “Borg, Omega, and Kubernetes” (free) | Kubernetes lineage and design | Depth |
| KubeCon / CloudNativeCon talks (free) | CNCF YouTube | Reference |
Practice, Tools, and Current Sources
The Kubernetes documentation (kubernetes.io/docs) is unusually high-quality for a technical project. The Concepts section explains what the system is doing rather than merely documenting API schemas; the Tasks section provides working examples; the Reference section is authoritative. Treat the conceptual documentation as primary material alongside the books.
kubectl is the primary interface to a Kubernetes cluster, and developing fluency with it is necessary for effective operation. The commands that reveal system state — kubectl get, kubectl describe, kubectl logs, kubectl events, kubectl top — are more informative than any dashboard for diagnosing specific problems. Running a local cluster with kind (Kubernetes in Docker) or minikube enables experimentation without a cloud provider account.
Lens or k9s (terminal-based Kubernetes UI) provide visual overviews of cluster state that kubectl alone does not. Lens is particularly useful for navigating across namespaces and for visualizing resource relationships; k9s is preferred by operators who work primarily in the terminal.
The CNCF Landscape (landscape.cncf.io) maps the cloud-native ecosystem. It is overwhelming as a starting point but useful for understanding the space of available tools within a given category once the foundational concepts are established.
Engineering blogs from organizations running cloud-native infrastructure at scale — Spotify, Pinterest, Airbnb, Shopify, Stripe — document production behaviors that books and documentation do not address. The gap between “Kubernetes works in principle” and “Kubernetes works in our production environment” is where these blogs live.
| Resource | Platform | Type |
|---|---|---|
| Kubernetes documentation (free) | kubernetes.io/docs | Reference |
| Kubernetes official tutorials (free) | kubernetes.io/docs/tutorials | Practice |
| Certified Kubernetes Administrator curriculum (exam paid) | cncf.io | Auxiliary |
| Kubernetes the Hard Way (free) | github.com/kelseyhightower | Practice |
| kind / minikube (free) | kind.sigs.k8s.io / minikube.sigs.k8s.io | Practice |
| kubectl (free) | kubectl.docs.kubernetes.io | Practice |
| CNCF Landscape (free) | landscape.cncf.io | Reference |
| Engineering blogs (Spotify, Shopify, Stripe) | Various | Reference |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Kubernetes as a magic black box | Kubernetes presents a powerful abstraction that can be adopted through tutorials without understanding what it does. Operational fluency built on YAML templates collapses when standard patterns do not address the problem: the engineer can deploy applications but cannot diagnose why a Deployment is failing, cannot reason about scheduling decisions, cannot extend the system with custom resources. | Engage with what Kubernetes actually does. Work through Kubernetes the Hard Way to understand the constituent components. Read the Borg and Kubernetes lineage papers. Treat the conceptual sections of the Kubernetes documentation as primary material, not supplementary reading. Diagnose at least one production problem by reading actual system state with kubectl describe and kubectl events rather than relying on dashboard summaries. |
| Over-engineering the substrate | Cloud-native infrastructure can be applied to problems that do not require it. Small organizations with simple workloads adopt Kubernetes when a single application server would have sufficed; projects commit to multi-cluster architectures before having the operational maturity to manage one cluster well. The result is infrastructure complexity disproportionate to the workload, consuming engineering effort better directed elsewhere. | Ask honestly whether the workload’s requirements justify the infrastructure’s complexity. Continuous deployment cadence, horizontal scalability, multi-service coordination, geographic distribution — these are legitimate justifications. Intellectual attraction to the technology is not. McKinley’s “Choose Boring Technology” essay articulates the cost of unjustified complexity. |
| YAML as the endpoint of infrastructure engineering | Cloud-native infrastructure descriptions can grow to thousands of lines without discipline, with configuration duplicated across resources, environment-specific differences scattered across files, and relationships among resources obscured. Engineers who treat YAML production as the engineering activity produce infrastructure that resists change. | Treat infrastructure descriptions as code requiring the same engineering discipline as application code: avoid duplication, use abstraction, review changes, maintain tests. Adopt templating and composition tools (Helm, Kustomize) early, before the descriptions have grown to the point where lack of discipline is paralyzing. |
| Treating portability as default | Kubernetes promises portability across cloud providers through a common API. The core API is consistent; the surrounding dependencies — managed databases, identity systems, storage classes, networking infrastructure, cloud-specific services that real applications need — are vendor-specific. Organizations that adopt Kubernetes primarily for portability often discover their workloads are deeply bound to vendor-specific services. | Be explicit about which portability is achieved. For each application, assess what would actually be required to move it across providers and whether that movement is realistic given its service dependencies. Design for the portability that is actually needed rather than assuming the Kubernetes API provides it automatically. |
| Letting infrastructure patterns drive application architecture | Cloud-native infrastructure makes certain patterns easy: microservices, service-to-service communication, dynamic scaling. Practitioners can come to treat “what the infrastructure makes easy” as “what the application should do,” producing too many services because microservices are the default pattern, too much inter-service communication because service decomposition was the design tool. | Treat infrastructure architecture and application architecture as distinct concerns. Infrastructure should serve the application’s genuine requirements, not determine them. Apply the infrastructure’s capabilities instrumentally to an architecture designed from the application’s actual business and technical requirements. |
| Neglecting security posture | Kubernetes default configurations are not secure: Pods can access the Kubernetes API, containers run as root by default, network communication is unrestricted within a namespace. Organizations that deploy Kubernetes with default configurations and focus exclusively on application functionality discover security gaps when they prepare for production or undergo security review. | Apply security hardening from the beginning: Pod Security Standards, network policies, RBAC role minimization, image scanning, secret management through a secrets operator rather than plaintext ConfigMaps. Treat the Kubernetes security hardening guide as a deployment checklist, not as optional hardening to apply later. |
中文
基础设施过去是通过指令来管理的：在这台机器上运行这个脚本，以这种方式配置这个服务，按这个顺序安装这个包。当出现问题时，operator 诊断问题，并发出更多指令来修正它。这个模型在小规模时可以理解。到了大规模——数百个服务、数千台机器、每天多次持续部署——它就变得不可维持。没有任何数量的人脑能够保存“什么东西正在什么地方运行”的完整心智模型。把软件送入生产环境的 imperative scripts，会因为机器先前状态不同而产生不同结果。失败需要人工干预，而人工干预跟不上失败发生的速度。
容器编排是这门学科对这个规模问题的回应。它中心处的转变听起来简单，却影响深远：基础设施应当通过“它应该是什么”来描述，而不是通过“如何让它变成这样”的步骤来指令化。编排系统会把被描述的期望状态与 cluster 的实际状态进行比较，并持续采取行动来缩小差距——启动那些应该运行但没有运行的 workloads，终止那些不应该运行却正在运行的 workloads，用健康实例替换失败实例。人类 operators 管理描述；系统管理 reconciliation。这不只是便利性问题；它是一种根本不同的运维模型，其中正确性由 desired state 定义，而收敛到这个状态是系统的责任，不再是 operator 的责任。
Container 是 container image 的一个运行实例——container image 是一种可移植、自包含的包，包含应用代码及其依赖、运行时库和配置。Containers 使用 Linux kernel features（namespaces 用于隔离，cgroups 用于资源限制），在共享操作系统 kernel 上运行多个隔离进程；它们比普通进程有更强隔离保证，又比虚拟机开销更低。Container image 把这一切捕捉为一个不可变、可版本化的 artifact，并且能在开发、测试和生产中以同样方式运行。不可变性是架构性的：经过测试的 image，就是生产中运行的 image，从而消除早期部署模型中困扰许久的 “works on my machine” 失败类别。
前置知识：操作系统（§4.2）——Linux namespaces、cgroups 和进程隔离，是 containers 所依赖的 kernel primitives。计算机网络（§4.3）——service discovery、load balancing 和 network policy 是核心编排关切。分布式系统（§4.6）——编排系统本身就是分布式系统；etcd，Kubernetes configuration store，是一个基于 Raft 的分布式 key-value store。
从 Borg 到 Kubernetes 与云原生生态
Google 的容器编排史比 Kubernetes 早十年。Borg 是 Google 内部 cluster management system，大约从 2003 年开始开发，并在 Kubernetes 发布前多年，就已经在大规模管理 Google 的生产 workloads。Borg 引入了后来 Kubernetes 公开化的思想：声明式 job specifications，一个把 desired state 与 actual state 进行 reconciliation 的中心 control plane，在共享物理机器上进行 resource-request-and-limit scheduling，以及把 container 作为部署单位。构建 Borg 的工程师把这些经验带到了 Kubernetes——其中也包括这样一个教训：最早的设计选择会约束系统数十年。
Docker 由 Solomon Hykes 于 PyCon 2013 推出，它不是 scheduler，也不是 orchestration system；它是一个让 Linux containers 变得可用的工具。Docker 之前，使用 Linux containers（LXC）需要很深的系统管理知识。Docker 提供了简单的 command-line interface、标准 container image format（基于 union filesystem 的 layers），以及用于共享 images 的 registry（Docker Hub）。推出后几个月内，Docker 就使基于 container 的部署被那些永远不会直接配置 LXC 的实践者掌握。其采用速度很快，影响也很深：container images 成为标准打包单位，而如何大规模编排 containers 也迅速变成紧迫问题。
Google 于 2014 年 6 月发布 Kubernetes，并将其开源，作为 Borg 的重新设计版本，同时纳入 Borg 和更灵活的后继者 Omega 的经验。Borg 拥有专有 APIs，并与 Google 内部基础设施深度耦合；Kubernetes 则被设计为运行在通用云基础设施之上，并提供开放、可扩展 APIs。Kubernetes 最初发布论文的三位作者——Brendan Burns、Brian Grant 和 David Oppenheimer——明确把它描述为分享 Google 大规模运行 containers 经验的载体。Kubernetes 于 2016 年捐赠给 Cloud Native Computing Foundation（CNCF），而 CNCF 成为云原生生态的组织中心。
CNCF 模型——一个 vendor-neutral foundation，用于治理开源基础设施项目——使生态系统可以在不受单一厂商控制的情况下发展。Container networking（Calico、Cilium、Flannel）、service mesh（Istio、Linkerd）、certificate management（cert-manager）、secrets management（HashiCorp Vault）、continuous delivery（Argo CD、Flux）、monitoring（Prometheus）、distributed tracing（Jaeger、Zipkin）以及许多其他关切下的项目，都在同一个 umbrella 下治理。CNCF Landscape 映射了云原生空间中超过一千个项目，它既反映出生态的丰富性，也反映出实践者面对的导航挑战：在功能相似的工具之间选择，需要理解那些无法从表面描述中看出的设计权衡。
Service mesh 大约在 2017 年出现，它处理的是一个具体编排缺口：在包含大量独立 services 的 cluster 中管理 service-to-service communication。Service mesh 会在每个 service instance 旁注入一个 proxy sidecar；sidecar 处理 TLS、load balancing、circuit breaking 和 telemetry，使这些关切可以在不修改 service code 的情况下跨 services 统一管理。Istio 建立在 Envoy proxy 之上，成为主导 service mesh；Linkerd 提供更轻量的替代方案。Sidecar model 会带来开销——每个 pod 旁边都有一个 proxy——在大规模下变得显著。eBPF-based alternatives（例如 Cilium，它在 kernel 中而不是 sidecars 中实现 service-mesh 功能）和 sidecarless mesh models（Istio Ambient mode）的出现，正在重构 service mesh 空间；但底层问题——在一个 polyglot、多服务 cluster 中管理 east-west service communication——仍然是核心编排关切。
GitOps 作为一种运维模型出现，完成了声明式基础设施管理。如果基础设施由版本控制中的文件描述，那么任意时间点的基础设施状态都可以由该时间点 repository 的状态推导出来。Argo CD 和 Flux 是 GitOps controllers，它们持续把 cluster 的 actual state 与 Git repository 中声明的 desired state 同步；当 repository 更新时自动应用 changes，并检测、修正 cluster actual state 与 declared state 之间的 drift。GitOps 改变了 audit trail：对生产基础设施的每次变更，都是 repository 中经过 review 和 approve 的 commit，并带有可追溯到项目起点的历史。它所支持的运维模型——operators review pull requests，而不是执行 scripts——相比 imperative deployment procedures，在安全性和透明性上都有显著改进。
最近的结构性变化，是云原生基础设施与 AI/ML workloads 的融合。GPU scheduling 需要 Kubernetes 最初并未设计的编排 primitives：GPUs 不是可以互换的资源，fractional GPU sharing 有正确性含义，而需要特定 interconnect topologies（NVLink、InfiniBand）的 training jobs，需要 generic bin-packing schedulers 无法提供的 placement logic。KubeRay（Kubernetes 上的 distributed Ray）、Kubeflow platform，以及各种 GPU-aware scheduler extensions（NVIDIA GPU Operator、custom scheduler plugins）代表了工程回应。Inference serving pattern——把 requests 路由到加载了正确 model 的 service instances，并进行合适 batching 和 memory management——正在以尚未完全解决的方式压迫 Kubernetes service model。AI workloads 正在产生第二波编排创新，其规模可与最初产生 Kubernetes 的那一波相比。
Reconciliation、Scheduling 与 Developer-Operator Boundary
Declarative State 与 Reconciliation Loop
每一种 Kubernetes resource——Pod、Deployment、Service、Ingress、ConfigMap、PersistentVolumeClaim——都是存储在 etcd 这个 cluster distributed configuration store 中的 desired state 声明。Controllers watch 这些 resources，检测 desired state 与 actual state 之间的差异，并采取行动缩小差距。Deployment controller 会注意到运行中 Pods 数量与声明的 replica count 不一致，并相应创建或终止 Pods。Scheduler watch 尚未调度的 Pods，并把它们放置到满足约束的 nodes 上。Kubelet watch 分配给自己 node 的 Pods，并启动或停止 containers，使其匹配声明。
Reconciliation loop 是云原生基础设施的架构原语。理解它，就是理解 Kubernetes。Controllers 不是运行一次的 scripts；它们持续运行，在每个相关事件之后重新评估 desired 与 actual state 之间的差异。这意味着系统会自我修正：如果一个 Pod 因 node failure 被删除，Deployment controller 会检测到差异并创建替代实例。如果一个 Service 的 endpoints 因 Pods 重启而变化，Endpoints controller 会相应更新 service routing。在多数失败场景中，系统无需人工干预就会收敛到 desired state。
Operator pattern 把 reconciliation 扩展到 application-specific operational concerns。Kubernetes Operator 是一个 custom controller，它管理 custom resource，并用应用特定语义扩展 Kubernetes API。一个 Elasticsearch Operator 知道如何安全地扩展 Elasticsearch cluster（在移除 nodes 前等待 shard rebalancing）、如何执行 rolling upgrades（一次更新一个 node，并在继续前验证健康状态），以及如何进行 coordinated backups。一个 database Operator 管理 failover、replica promotion 和 backup scheduling。Operators 把原本需要人类专家掌握的运维知识编码下来，使其可自动化、可重复。Operator pattern 是 CNCF 生态中包含数百个 stateful applications operators 的原因；它也是在 Kubernetes 上运行 stateful workloads 比早期实践者预期更可处理的原因。
Scheduling：Placement Decisions 及其后果
Kubernetes scheduler 的工作，是把每个尚未调度的 Pod 分配到一个满足其约束的 node 上。这些约束来自许多来源：resource requests（Pod 所需的 CPU 和 memory）、node affinity 和 anti-affinity rules（Pod 偏好或要求的 node labels，或它应当靠近或远离的其他 Pods）、topology spread constraints（为了 resilience，把 Pods 分布到不同 zones 或 nodes 上），以及 pod disruption budgets（限制 voluntary disruptions 期间可同时不可用的 Pods 数量）。
Resource requests 和 limits 是对多数 workloads 来说最有后果的 scheduling parameters。Pod 的 resource request 声明它需要什么；scheduler 会把它放到有足够可用容量的 node 上。Pod 的 resource limit 声明它最多能用多少；kubelet 会通过 CPU throttling，以及对超过 memory limits 的进程执行 OOM-killing 来强制执行。Request 和 limit 之间的差值是 slack——当 node 有闲置容量时，允许 Pods 突破其 requested resources 的 headroom。正确设置 resource requests，是运行 Kubernetes 的实践难题之一：requests 太低会导致 Pods 被调度到 overcommitted nodes 上，并在负载下退化；requests 太高会阻碍高效 bin-packing，浪费 cluster resources。
Multi-tenant clusters 中的 workload isolation 要求理解一点：Kubernetes namespace boundaries 并不提供强安全隔离——同一 node 上不同 namespaces 中的 Pods 共享 kernel，并且可能彼此干扰。强隔离要求 network policies（限制哪些 Pods 可以通信）、Pod Security Standards（强制 security contexts，限制 privilege escalation），或物理隔离（独立 node pools 或独立 clusters）。Kubernetes security model 是分层关切，而不是单一控制。
Developer-Operator Interface
云原生基础设施把抽象插入到 application developers 与 operational substrate 之间。Developers 声明应用需求（resource requirements、configuration、health endpoints、scaling behavior）；platform 提供执行环境。这种抽象允许 developers 在没有深厚基础设施知识的情况下部署应用，也允许 operators 演化基础设施，而不破坏应用声明。
抽象设计问题——哪些关切暴露给 developers，哪些隐藏，如何允许运维团队配置 defaults——是 platform engineering 的中心挑战。一个暴露太多基础设施复杂性的 platform，会迫使 developers 理解与其工作无关的细节。一个隐藏太多的 platform，又会阻止 developers 表达基础设施本可以满足的要求。Platform engineering 作为一门纪律，会把 deliberate product thinking 应用于 internal developer platform 的设计：定义 user personas，理解 developer workflows，建立 feedback loops，并基于 developer experience 演化 platform。
Helm 是 Kubernetes package manager，它通过引入 templating 处理 YAML complexity 问题。一个 Helm chart 会把相关 Kubernetes resources 与 configurable parameters 打包在一起，使同一个 chart 可以用不同配置在 development 和 production 中部署 Redis instance。Kustomize 则采取不同方式——不使用 templating，而采用 overlay-based configuration——在保留 YAML 简洁性的同时，允许 environment-specific customization，而不重复 base configuration。二者都处理了 raw YAML 在规模化后变得不可维护的实践问题；二者之间的选择，反映了对复杂性管理的不同偏好。
学习这一部分会改变什么
容器编排会改变实践者思考部署和运维可靠性的方式。
第一个变化，是能够以运维方式推理基础设施，而不只是部署它。这意味着能够诊断为什么一个 Pod 处于 pending 状态（node resources 不足、scheduler constraints 被违反、node affinity 不匹配），为什么一个 Service 没有 routing（endpoint selection failures、network policy blocks、readiness probe failures），为什么 Deployment rollout 停滞（maxUnavailable 和 maxSurge 设置、readiness gates、pod disruption budget violations）。这种诊断能力要求理解 orchestration system 内部正在做什么，而不只是知道如何向它表达 desired state。
第二个变化，是判断什么时候 cloud-native infrastructure 合适。对于那些需要 horizontal scalability、continuous deployment、multi-service coordination 和 geographic distribution 的 workloads，这门纪律提供真实能力。对于不具备这些要求的 workloads，它也会施加真实开销——运维复杂性、团队专业能力要求、基础设施成本。真正内化这门纪律的实践者，能够诚实评估哪些特征足以证明投入合理，而不是因为 Kubernetes 是当前标准就默认采用它。
第三个变化，是带着部署基底来设计应用。Cloud-native applications 是为 ephemerality 设计的：containers 可能在任何时候被移动或重启，因此 application state 必须 externalized（存到 databases、object storage 或 distributed caches），而不是保存在本地。它们暴露 health endpoints（liveness 和 readiness probes），供 orchestrator 路由流量和重启不健康实例。它们发出适合 distributed observation 的 structured telemetry。这些要求来自 orchestration system 的设计；违反这些要求的应用，会产生 orchestration system 无法自动恢复的失败模式。
资源
书籍与文本
Hightower、Burns 和 Beda 的 Kubernetes: Up and Running（第 3 版，O’Reilly，2022）是经典入门。三位作者都是 Kubernetes 早期发展中最重要的工程师之一，这本书按照设计者意图来呈现系统——覆盖 Pods、Services、Deployments 和主要 workload types，并解释每个概念是什么、为什么存在。其逐步构建方法让系统变得可理解，而不是让人记忆一组 YAML schemas。
Burns 的 Designing Distributed Systems（O’Reilly，2018）覆盖云原生基础设施使之可实践的分布式系统模式——sidecars、ambassadors、leader election、work queues、scatter-gather——并面向实践者提供说明。它提供云原生应用设计的词汇。Ibryam 和 Huss 的 Kubernetes Patterns（第 2 版，O’Reilly，2023）专门覆盖云原生应用的 recurring design patterns——如何在 Kubernetes 约束下组织 health probes、configuration injection、leader election 及其他关切。Davis 的 Cloud Native Patterns 更宽泛地处理应用架构。
对于生产运维，Production Kubernetes（Vyas、Love、Heller、Wood，O’Reilly，2021）覆盖运行自有 clusters 的组织所需的 security hardening、multi-tenancy 和 operational tooling。对于 infrastructure as code，Brikman 的 Terraform: Up and Running（第 3 版，O’Reilly，2022）是标准 Terraform 入门。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Hightower, Burns & Beda, Kubernetes: Up and Running（第 3 版） | 经典入门 | 入门 |
| Burns, Designing Distributed Systems | 云原生架构模式 | 入门 |
| Ibryam & Huss, Kubernetes Patterns（第 2 版） | 设计模式参考 | 参考 |
| Vyas et al., Production Kubernetes | 生产运维深度 | 深入 |
| Davis, Cloud Native Patterns | 云原生应用架构 | 深入 |
| Brikman, Terraform: Up and Running（第 3 版） | Infrastructure as code | 入门 |
| Skelton & Pais, Team Topologies | Platform engineering 的组织基础 | 深入 |
课程、论文与当前资料
Hightower 的 Kubernetes the Hard Way（可在 github.com/kelseyhightower/kubernetes-the-hard-way 免费在线获取）会引导学习者从组成组件组装一个 Kubernetes cluster——API server、controllers、scheduler、kubelets、networking、storage。这个练习要求很高，并能产生书本无法提供的理解：亲手组装过 cluster 的工程师，知道 managed services 隐藏了什么，也能在出问题时推理这些内容。它是必要的深度练习，最好在读完 Kubernetes: Up and Running 后进行。
奠基论文是 Verma 等人的 “Large-scale cluster management at Google with Borg”（EuroSys 2015，免费）和 Burns 等人的 “Borg, Omega, and Kubernetes”（ACM Queue 2016，免费）。二者共同解释了 Kubernetes 从哪里来，以及其创造者从 Google 生产经验中带来了哪些设计教训。阅读它们会让系统更可读——许多看似任意的 Kubernetes 设计选择，其实都是对 Borg 经验的回应，而论文记录了这些回应。
KubeCon 和 CloudNativeCon talks（YouTube 免费）代表了这个领域当前的实践者状态。CNCF YouTube channel 存档了每年会议的 talks；按主题过滤（security、networking、platform engineering、AI/ML workloads）可以得到对当前关切的实践覆盖。
| 课程/资源 | 平台 | 类型 |
|---|---|---|
| Verma et al., “Large-scale cluster management at Google with Borg”（免费） | Borg 架构论文 | 深入 |
| Burns et al., “Borg, Omega, and Kubernetes”（免费） | Kubernetes 谱系与设计 | 深入 |
| KubeCon / CloudNativeCon talks（免费） | CNCF YouTube | 参考 |
实践、工具与当前资料
Kubernetes documentation（kubernetes.io/docs）作为技术项目文档，质量异常高。Concepts 部分解释系统正在做什么，而不只是记录 API schemas；Tasks 部分提供可运行例子；Reference 部分具有权威性。应把概念文档当作与书籍并列的一手材料。
kubectl 是 Kubernetes cluster 的主要接口，熟练使用它是有效运维的必要条件。揭示系统状态的命令——kubectl get、kubectl describe、kubectl logs、kubectl events、kubectl top——在诊断具体问题时，比任何 dashboard 都更有信息量。使用 kind（Kubernetes in Docker）或 minikube 运行本地 cluster，可以在没有云服务账号的情况下进行实验。
Lens 或 k9s（terminal-based Kubernetes UI）提供 kubectl 单独无法提供的 cluster state 可视化概览。Lens 对跨 namespaces 导航和可视化 resource relationships 尤其有用；k9s 则更受主要在 terminal 中工作的 operators 偏好。
CNCF Landscape（landscape.cncf.io）映射了云原生生态。作为起点，它令人不堪重负；但在基础概念建立之后，它很适合用来理解某个具体类别中可用工具的空间。
运行大规模云原生基础设施的组织工程博客——Spotify、Pinterest、Airbnb、Shopify、Stripe——记录了书籍和文档不会处理的生产行为。“Kubernetes 原理上可行”与“Kubernetes 在我们的生产环境中可行”之间的差距，正是这些博客所处的位置。
| 资源 | 平台 | 类型 |
|---|---|---|
| Kubernetes documentation（免费） | kubernetes.io/docs | 参考 |
| Kubernetes official tutorials（免费） | kubernetes.io/docs/tutorials | 实践 |
| Certified Kubernetes Administrator curriculum（考试付费） | cncf.io | 辅助 |
| Kubernetes the Hard Way（免费） | github.com/kelseyhightower | 实践 |
| kind / minikube（免费） | kind.sigs.k8s.io / minikube.sigs.k8s.io | 实践 |
| kubectl（免费） | kubectl.docs.kubernetes.io | 实践 |
| CNCF Landscape（免费） | landscape.cncf.io | 参考 |
| Engineering blogs（Spotify、Shopify、Stripe） | Various | 参考 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 把 Kubernetes 当成魔法黑箱 | Kubernetes 提供强大抽象，人们可以通过教程采用它，而不理解它实际做了什么。建立在 YAML templates 上的运维熟练度，一旦标准模式无法处理问题就会崩塌：工程师能部署应用，却无法诊断 Deployment 为什么失败，无法推理 scheduling decisions，也无法用 custom resources 扩展系统。 | 接触 Kubernetes 实际做的事情。完成 Kubernetes the Hard Way，理解其组成组件。阅读 Borg 与 Kubernetes 谱系论文。把 Kubernetes 文档中的 concepts 部分当作一手材料，而不是补充阅读。至少诊断一次生产问题，使用 kubectl describe 和 kubectl events 阅读实际系统状态，而不是依赖 dashboard 摘要。 |
| 过度工程化底层基底 | Cloud-native infrastructure 可以被应用到并不需要它的问题上。小组织和简单 workloads 在一台 application server 就足够时采用 Kubernetes；项目在还没有运维成熟度管理好一个 cluster 之前，就承诺 multi-cluster architectures。结果是与 workload 不成比例的基础设施复杂性，消耗了本可以用于其他地方的工程精力。 | 诚实询问 workload 的要求是否足以证明基础设施复杂性合理。Continuous deployment cadence、horizontal scalability、multi-service coordination、geographic distribution——这些是合理理由。对技术本身的智识吸引不是理由。McKinley 的 “Choose Boring Technology” 文章说明了不合理复杂性的成本。 |
| 把 YAML 当成基础设施工程的终点 | 如果缺乏纪律，cloud-native infrastructure descriptions 可以增长到数千行，configuration 在 resources 之间重复，环境差异散落在文件中，resources 之间的关系变得模糊。把生产 YAML 当成工程活动本身的工程师，会制造出抗拒变化的基础设施。 | 把 infrastructure descriptions 当作代码，并施加与 application code 相同的工程纪律：避免重复，使用抽象，review changes，维护 tests。尽早采用 templating 和 composition tools（Helm、Kustomize），不要等 descriptions 增长到缺乏纪律已经使人瘫痪时才处理。 |
| 把 portability 当成默认结果 | Kubernetes 通过通用 API 承诺跨 cloud providers 的 portability。核心 API 是一致的；但周边依赖——managed databases、identity systems、storage classes、networking infrastructure、真实应用所需的 cloud-specific services——都是 vendor-specific 的。主要为了 portability 而采用 Kubernetes 的组织，常常发现自己的 workloads 深度绑定到 vendor-specific services。 | 明确说明实现的是哪种 portability。对每个应用，评估把它跨 providers 移动到底需要什么，以及考虑到其 service dependencies，这种迁移是否现实。为实际需要的 portability 设计，而不是假定 Kubernetes API 会自动提供它。 |
| 让基础设施模式驱动应用架构 | Cloud-native infrastructure 使某些模式变得容易：microservices、service-to-service communication、dynamic scaling。实践者可能把“基础设施使什么容易”误当成“应用应该做什么”，因为 microservices 是默认模式而制造过多 services，因为 service decomposition 是设计工具而制造过多 inter-service communication。 | 把 infrastructure architecture 和 application architecture 当作不同关切。基础设施应服务于应用的真实需求，而不是决定应用需求。根据应用真实的业务和技术要求设计架构，再工具性地使用基础设施能力。 |
| 忽视安全态势 | Kubernetes 默认配置并不安全：Pods 可以访问 Kubernetes API，containers 默认以 root 运行，namespace 内网络通信不受限制。只关注应用功能、用默认配置部署 Kubernetes 的组织，会在准备生产或接受安全审查时发现安全缺口。 | 从一开始就应用 security hardening：Pod Security Standards、network policies、RBAC role minimization、image scanning、通过 secrets operator 而不是 plaintext ConfigMaps 管理 secrets。把 Kubernetes security hardening guide 当作部署 checklist，而不是之后可选应用的 hardening。 |