English

Container orchestration manages the placement and lifecycle of workloads across machines. If a service should have three replicas and one disappears, a controller can request a replacement. The service description remains, while the particular processes and machines can change. This separates the desired arrangement from the sequence of actions used to reach it.

A container image packages a filesystem and execution metadata. A running Linux container normally consists of processes using namespaces, cgroups, and other isolation and resource controls while sharing the host kernel. An image reduces variation in packaged software, but behavior still depends on the kernel, CPU, configuration, permissions, network, and external services. Pinning an image by digest identifies the artifact; a mutable tag alone does not.

Orchestration automates repeated operational work; it does not define application correctness. Replacing a failed process cannot reconstruct data that was never persisted, and starting two workers can duplicate an action unless the application controls ownership or retries. Learn the boundary between what the platform manages and what the application must establish.

Prerequisites: Operating systems (§4.2), computer networks (§4.3), and distributed systems (§4.6). Linux process isolation and replicated control-plane state connect these topics to orchestration.

From Borg to Kubernetes and the Cloud-Native Ecosystem

Shared-machine isolation predates modern container tools: Unix chroot, FreeBSD jails, and Linux namespaces and cgroups addressed different parts of the problem. Packaging, isolation, scheduling, and recovery are separate capabilities that later platforms combined. Their history explains why a container is not simply a smaller virtual machine.

Google’s Borg managed large clusters using job specifications, scheduling, resource management, and recovery. Its experience showed that cluster management needed to account for both service reliability and efficient sharing of machines. Kubernetes, announced in 2014, was a new project informed by Borg and Omega, with APIs intended for use beyond Google’s internal environment.

Docker, introduced in 2013, made building, distributing, and running container images more accessible. Reproducible packaging helped teams move the same artifact between environments, while registries supported distribution. This solved a different part of the problem from deciding where thousands of workloads should run.

Kubernetes became CNCF’s first hosted project in 2016, following the foundation’s establishment in 2015. Its extensible APIs enabled separate projects to supply networking, storage, deployment, and observation capabilities. Appearing in a cloud-native landscape does not imply that every project shares the same foundation, governance, or integration contract. Select components for a required capability and understand who maintains the boundary between them.

Service meshes moved selected communication functions into a shared infrastructure layer: identity, transport protection, traffic policy, and telemetry. Sidecar proxies are one implementation; other designs use node-level components or combinations of kernel and userspace mechanisms. A mesh still needs resources and correct configuration, and it does not decide which business operation a user may perform.

GitOps controllers compare deployed resources with a versioned declaration and attempt to reconcile differences. Version control can record reviewed changes to that declaration. It does not contain all database state, external secrets, runtime discoveries, or emergency actions, nor does it force every commit to have been reviewed. Reproducible deployment also needs pinned dependencies, controlled access, and a recovery procedure.

Accelerator workloads add placement constraints such as device type, memory, topology, and coordination among job workers. Device plugins expose resources; operators may install drivers and manage related components; schedulers make placement decisions. These roles are related but distinct. The same general question remains: what information does the scheduler need to place this workload usefully?

Reconciliation, Scheduling, and the Developer-Operator Boundary

Reconciliation maintains a declared arrangement, scheduling chooses a location, and the application defines useful behavior. Understanding their interfaces is more useful than treating the platform as one automatic recovery mechanism.

Declarative State and the Reconciliation Loop

A Pod groups one or more containers that share selected resources, including a network namespace, and are scheduled together. The API server exposes resource objects; etcd normally persists control-plane state. A Deployment manages ReplicaSets, which manage Pods. The scheduler assigns an unscheduled Pod to a suitable node, and that node’s kubelet works with the container runtime to run its containers. These cooperating components do not perform one atomic cluster-wide action.

Controllers repeatedly compare observed state with desired state and take steps to reconcile them. Observations can be stale, actions can fail, and events can repeat, so reconciliation must tolerate retries. If a declaration requests three replicas but no node has capacity, the declaration can remain valid while the workload stays pending. Desired state is a target, not evidence that it has been achieved.

A disconnected node may still be running a workload while the control plane considers it unavailable. Replacing that workload therefore does not provide exactly-once execution. Applications that require exclusive access to a resource may need fencing, storage coordination, or a lease protocol with correctly enforced ownership.

Operators add application-specific controllers, often with custom resources. They can encode procedures for backup, upgrade, and recovery, but their guarantees depend on their implementation and the application. A StatefulSet provides stable identities and supports persistent storage association; it does not itself implement database replication, transaction recovery, or backups.

Scheduling: Placement Decisions and Their Consequences

Scheduling filters nodes against requirements and then scores feasible placements. Resource requests, affinity rules, available devices, and topology constraints affect placement. Spreading replicas across failure domains can reduce correlated loss, but only if the supposedly separate domains do not share the relevant dependency.

For CPU and memory, requests inform placement using declared allocation rather than simply observing current utilization. Limits constrain runtime usage: CPU limits can cause throttling, while memory limits are enforced through memory controls that can lead to an out-of-memory kill. A CPU request of one core does not necessarily mean exclusive access to a particular physical core. The resource-management documentation explains these different roles.

A PodDisruptionBudget limits certain voluntary disruptions through the eviction API; it is not an ordinary node-placement constraint or a guarantee against machine failure. Deployment rolling updates use their own rollout settings and are not restricted by PDBs in the same way. This distinction matters when planning maintenance; see the disruption model.

Isolation combines identity, authorization, network controls, workload privileges, resource controls, and appropriate machine boundaries. A namespace is an organizational and policy scope, not a separate kernel. A network policy controls selected traffic when enforced by the networking implementation; it does not provide virtual-machine isolation. Choose the boundary according to the workloads and failure model.

Autoscaling changes desired capacity using observed signals, then waits for scheduling, image availability, startup, and readiness. Adding replicas cannot fix every bottleneck, such as one saturated database. CPU utilization relative to requests and queue length answer different questions, so choose a signal that represents the workload’s capacity requirement.

The Developer-Operator Interface

Cloud-native infrastructure interposes abstractions between application developers and the operational substrate. Developers declare their application’s needs (resource requirements, configuration, health endpoints, scaling behavior); the platform provides the execution environment. This abstraction allows developers to deploy applications without deep infrastructure knowledge, and it allows operators to evolve the infrastructure without breaking application declarations.

The abstraction design question — which concerns to expose to developers, which to hide, how to allow operational teams to configure defaults — is the central challenge of platform engineering. A platform that exposes too much infrastructure complexity forces developers to understand details irrelevant to their work. A platform that hides too much prevents developers from expressing requirements the infrastructure can satisfy. Platform engineering, as a discipline, applies deliberate product thinking to the design of the internal developer platform: defining user personas, understanding developer workflows, building feedback loops, and evolving the platform based on developer experience.

Helm, the Kubernetes package manager, addresses the YAML complexity problem by introducing templating. A Helm chart packages related Kubernetes resources with configurable parameters, allowing a single chart to deploy a Redis instance with different configurations in development and production. Kustomize takes a different approach — overlay-based configuration without templating — that preserves YAML’s simplicity while allowing environment-specific customization without duplicating base configuration. Both address the practical problem that raw YAML at scale becomes unmaintainable; the choice between them reflects preferences about complexity management.

Health checks are part of this interface. Readiness indicates whether an instance should receive normal service traffic; liveness can trigger restart; startup probes allow initialization before other probes apply. Restarting a healthy process merely because a shared dependency is down can amplify an outage. A useful probe distinguishes a condition that restart can repair from one that calls for waiting, degraded service, or a different intervention.

A rollout can keep old and new versions running together. API changes, message formats, and database schema changes must therefore tolerate that overlap. Rolling back an image does not undo a destructive schema change or an already issued external action. Deployment design includes data and protocol compatibility, not only container replacement.

What Studying This Changes

Container orchestration changes how practitioners think about deployment and operational reliability.

Diagnose one boundary at a time: whether a Pod was admitted, scheduled, started, and marked ready; whether service selection found it; and whether traffic could reach it. A pending Pod, a crashing container, and an unreachable ready endpoint call for different evidence. Events, status conditions, logs, and application telemetry help locate which transition failed.

Evaluate orchestration by the work it removes and the responsibilities it adds: deployment, replacement, access control, upgrades, and cluster recovery. A small service may need a simpler platform, while a larger fleet may benefit from shared controllers and policy. The decision depends on workload and operating capacity, not a universal minimum or maximum service count.

Design application state according to its lifetime. Disposable caches may be local; durable records need storage and recovery suited to their failure model. Persistent volumes are one mechanism, while external databases or object stores are others. Test replacement and recovery explicitly, including the case where the old instance has not actually stopped.

Resources

Books and Texts

Kubernetes: Up and Running introduces workload objects and their relationships. Designing Distributed Systems, Kubernetes Patterns, and Cloud Native Patterns explain recurring application structures. Production Kubernetes studies operation and tenancy; Terraform: Up and Running addresses infrastructure provisioning; Team Topologies adds organizational design. These resources solve different problems, so select the relevant part instead of treating the whole list as a prerequisite.

Book Role Type
Hightower, Burns & Beda, Kubernetes: Up and Running (3rd ed.) Kubernetes objects and operation Entry
Burns, Designing Distributed Systems Cloud-native architectural patterns Entry
Ibryam & Huss, Kubernetes Patterns (2nd ed.) Design patterns reference Reference
Vyas et al., Production Kubernetes Production operation depth Depth
Davis, Cloud Native Patterns Application architecture for cloud-native Depth
Brikman, Terraform: Up and Running (3rd ed.) Infrastructure as code Entry
Skelton & Pais, Team Topologies Platform engineering organizational foundation Depth

Courses, Papers, and Current Sources

The Borg paper and Borg, Omega, and Kubernetes explain cluster-management requirements and design history. Kubernetes the Hard Way exposes components usually hidden by managed services; use its stated environment and version assumptions. KubeCon / CloudNativeCon talks provide topic-specific experience. Compare each talk’s scale and constraints with your own question.

Course/Resource Platform Type
Verma et al., “Large-scale cluster management at Google with Borg” (free) Borg architecture paper Depth
Burns et al., “Borg, Omega, and Kubernetes” (free) Kubernetes lineage and design Depth
KubeCon / CloudNativeCon talks (free) CNCF YouTube Reference

Practice, Tools, and Current Sources

Use a local kind or minikube cluster to inspect Pods, controller status, scheduling events, logs, and readiness. kubectl, k9s, and Lens offer different views of cluster state; no interface replaces understanding the underlying resource. Follow the Kubernetes Concepts and Tasks documentation while changing one condition at a time. The CNCF Landscape is a category map, not a requirement to install every adjacent tool.

Resource Platform Type
Kubernetes documentation (free) kubernetes.io/docs Reference
Kubernetes official tutorials (free) kubernetes.io/docs/tutorials Practice
Certified Kubernetes Administrator curriculum (exam paid) cncf.io Auxiliary
Kubernetes the Hard Way (free) github.com/kelseyhightower Practice
kind / minikube (free) kind.sigs.k8s.io / minikube.sigs.k8s.io Practice
kubectl (free) kubectl.docs.kubernetes.io Practice
CNCF Landscape (free) landscape.cncf.io Reference
Engineering blogs (Spotify, Shopify, Stripe) Various Reference

Traps

Trap Why it misleads Better response
Treating Kubernetes as one black box Admission, scheduling, startup, readiness, and routing can fail separately. Inspect the status and evidence for the transition that failed.
Adding infrastructure before establishing a requirement Operational complexity can outweigh automation benefits for a small workload. Compare a simpler deployment against the actual scaling, recovery, and delivery needs.
Treating YAML volume as progress More configuration can hide relationships and inconsistent defaults. Keep declarations reviewable and validate rendered configuration against intended behavior.
Assuming a shared API guarantees portability Storage, identity, networking, and managed-service dependencies can remain provider-specific. List the dependencies and test the migration that actually matters.
Letting orchestration choose application boundaries Easy deployment of many services does not establish a useful decomposition. Choose service and data boundaries from application requirements and coordination costs.
Assuming namespace separation is sufficient security Permissions, credentials, network policy, workload privilege, and shared kernels have separate effects. Inspect the actual controls and use isolation suited to the tenant and workload model.

中文

容器编排管理跨机器工作负载的放置与生命周期。如果一个服务应该有三个副本,而其中一个消失,控制器可以请求创建替代副本。服务描述保留下来,具体进程与机器则可以变化。这把期望的组织方式与达到它所需的动作序列分开。

容器镜像打包文件系统与执行元数据。运行中的 Linux 容器通常由使用命名空间、cgroup 等隔离与资源控制机制的进程组成,并共享宿主机内核。镜像减少所打包软件的差异,但行为仍依赖内核、CPU、配置、权限、网络与外部服务。用摘要固定镜像可以确定具体制品;仅靠可变标签不行。

编排自动化重复的运维工作,但不定义应用正确性。替换失败进程不能重建从未持久化的数据;启动两个工作进程,也可能重复执行动作,除非应用控制所有权或重试。需要学清平台管理什么,以及应用必须自己建立什么性质。

前置知识:操作系统(§4.2)、计算机网络(§4.3)与分布式系统(§4.6)。Linux 进程隔离与控制平面状态复制,把这些主题连接到编排。

从 Borg 到 Kubernetes 与云原生生态

共享机器上的隔离早于现代容器工具:Unix chroot、FreeBSD jail、Linux 命名空间与 cgroup 分别处理其中不同部分。打包、隔离、调度与恢复是不同能力,后来的平台把它们组合起来。这段历史解释了为什么容器不能简单理解为更小的虚拟机。

Google 的 Borg 使用任务描述、调度、资源管理与恢复机制管理大型集群。其经验表明,集群管理需要同时考虑服务可靠性与机器的高效共享。2014 年公布的 Kubernetes 是吸收 Borg 与 Omega 经验的新项目,其 API 面向 Google 内部环境之外的使用者。

2013 年推出的 Docker,使构建、分发与运行容器镜像更容易。可复现的软件打包帮助团队在环境之间移动同一制品,镜像仓库支持分发。这与决定成千上万个工作负载应该运行在哪里,解决的是不同部分的问题。

CNCF 于 2015 年成立,Kubernetes 在 2016 年成为其首个托管项目。可扩展 API 使不同项目能够提供网络、存储、部署与观测能力。出现在云原生项目图谱中,并不表示每个项目具有相同基金会、治理或整合约定。应按所需能力选择组件,并理解谁负责维护组件之间的接口。

服务网格把选定的通信功能放入共享基础设施:身份、传输保护、流量策略与遥测。边车代理是一种实现,其他设计使用节点级组件,或组合内核与用户态机制。网格仍需要资源与正确配置,也不会替应用决定用户可以执行哪种业务操作。

GitOps 控制器把已部署资源与经过版本管理的声明比较,并尝试消除差异。版本控制可以记录声明中经过审查的变更,但不包含全部数据库状态、外部秘密信息、运行时发现或紧急操作,也不强制每次提交都经过审查。可复现部署还需要固定依赖、控制访问并具备恢复程序。

加速器工作负载增加了设备类型、内存、拓扑与任务工作进程之间协调等放置约束。设备插件暴露资源,Operator 可以安装驱动并管理相关组件,调度器作出放置决定。这些角色相关,但各不相同。一般性问题仍然是:调度器需要什么信息,才能把这个工作负载放到合适的位置?

Reconciliation、Scheduling 与 Developer-Operator Boundary

协调循环维持声明的组织方式,调度选择位置,应用定义有用的行为。理解它们的接口,比把整个平台视为一个自动恢复机制更有帮助。

Declarative State 与 Reconciliation Loop

Pod 把一个或多个容器分为一组,共享网络命名空间等选定资源,并一起调度。API 服务器提供资源对象接口,etcd 通常持久保存控制平面状态。Deployment 管理 ReplicaSet,后者管理 Pod。调度器把尚未调度的 Pod 分配给合适节点,该节点的 kubelet 再与容器运行时协作启动容器。这些组件的协作并非一次覆盖整个集群的原子动作。

控制器反复比较观察状态与期望状态,并逐步消除差异。观察可能陈旧,动作可能失败,事件可能重复,因此协调循环必须容忍重试。如果声明要求三个副本,但没有节点具有容量,声明可以有效,工作负载却持续等待。期望状态是目标,不是目标已经实现的证据。

控制平面认为某个断联节点不可用时,该节点上的工作负载可能仍在运行。因此,替换工作负载不提供恰好一次执行。需要独占资源的应用,可能需要栅栏机制、存储协调,或正确执行所有权限制的租约协议。

Operator 增加应用专用控制器,通常配合自定义资源。它可以编码备份、升级与恢复程序,但保证依赖具体实现和应用。StatefulSet 提供稳定标识,并支持与持久存储关联;它本身不实现数据库复制、事务恢复或备份。

Scheduling:Placement Decisions 及其后果

调度先按要求过滤节点,再对可行放置评分。资源请求、亲和性规则、可用设备与拓扑约束影响放置。把副本分散到不同故障域,可以减少相关故障造成的损失,但前提是这些被视为独立的域没有共享相关依赖。

对 CPU 与内存,request 根据声明的资源分配参与放置,而非只观察当前利用率。limit 约束运行时使用:CPU limit 可以导致限流,内存 limit 通过内存控制执行,可能导致内存不足终止。请求一个 CPU 核心,不必然意味着独占某个物理核心。资源管理文档解释了这些不同职责。

PodDisruptionBudget 通过驱逐 API 限制特定的自愿中断,不是普通的节点放置约束,也不保证抵御机器故障。Deployment 滚动更新使用自身的发布设置,不以相同方式受 PDB 限制。规划维护时,这个区分很重要,可参见中断模型。

隔离结合身份、授权、网络控制、工作负载权限、资源控制与适当的机器边界。命名空间是组织和策略范围,不是独立内核。网络策略在网络实现执行它时控制选定流量,并不提供虚拟机隔离。应按工作负载与故障模型选择边界。

自动扩缩容使用观察到的信号改变期望容量,随后还要等待调度、镜像可用、启动与就绪。增加副本不能解决所有瓶颈,例如一个已经饱和的数据库。相对 request 的 CPU 利用率与队列长度回答不同问题,因此应选择能表达工作负载容量需求的信号。

Developer-Operator Interface

云原生基础设施把抽象插入到 application developers 与 operational substrate 之间。Developers 声明应用需求(resource requirements、configuration、health endpoints、scaling behavior);platform 提供执行环境。这种抽象允许 developers 在没有深厚基础设施知识的情况下部署应用,也允许 operators 演化基础设施,而不破坏应用声明。

抽象设计问题——哪些关切暴露给 developers,哪些隐藏,如何允许运维团队配置 defaults——是 platform engineering 的中心挑战。一个暴露太多基础设施复杂性的 platform,会迫使 developers 理解与其工作无关的细节。一个隐藏太多的 platform,又会阻止 developers 表达基础设施本可以满足的要求。Platform engineering 作为一门纪律,会把 deliberate product thinking 应用于 internal developer platform 的设计:定义 user personas,理解 developer workflows,建立 feedback loops,并基于 developer experience 演化 platform。

Helm 是 Kubernetes package manager,它通过引入 templating 处理 YAML complexity 问题。一个 Helm chart 会把相关 Kubernetes resources 与 configurable parameters 打包在一起,使同一个 chart 可以用不同配置在 development 和 production 中部署 Redis instance。Kustomize 则采取不同方式——不使用 templating,而采用 overlay-based configuration——在保留 YAML 简洁性的同时,允许 environment-specific customization,而不重复 base configuration。二者都处理了 raw YAML 在规模化后变得不可维护的实践问题;二者之间的选择,反映了对复杂性管理的不同偏好。

健康检查是这一接口的一部分。就绪检查表示实例是否应该接收正常服务流量,存活检查可以触发重启,启动检查则为初始化提供时间,再启用其他检查。仅仅因为共享依赖不可用,就重启正常进程,可能扩大故障。有用的检查应区分重启能修复的情况,以及需要等待、降级服务或其他干预的情况。

发布过程中,新旧版本可能同时运行。因此,API 变更、消息格式与数据库模式变更需要容忍这种重叠。回退镜像不会撤销破坏性模式变更或已经执行的外部动作。部署设计包括数据与协议兼容性,而不只是替换容器。

学习这一部分会改变什么

容器编排会改变实践者思考部署和运维可靠性的方式。

逐个边界诊断:Pod 是否被接纳、调度、启动并标记就绪;服务选择是否找到它;流量能否到达它。等待中的 Pod、反复崩溃的容器与不可达的就绪端点,需要不同证据。事件、状态条件、日志与应用遥测可以帮助确定哪次转换失败。

评估编排时,应比较它减少的工作与增加的责任:部署、替换、访问控制、升级和集群恢复。小型服务可能适合更简单的平台,大型服务群则可能受益于共享控制器和策略。决策依赖工作负载与运维能力,不存在通用的最小或最大服务数量界限。

按生命周期设计应用状态。可丢弃缓存可以保存在本地,持久记录则需要符合故障模型的存储与恢复方式。持久卷是一种机制,外部数据库或对象存储是其他机制。应明确测试替换与恢复,包括旧实例实际上尚未停止的情况。

资源

书籍与文本

Kubernetes: Up and Running 介绍工作负载对象及其关系。Designing Distributed Systems、Kubernetes Patterns 与 Cloud Native Patterns 解释反复出现的应用结构。Production Kubernetes 研究运行与租户管理,Terraform: Up and Running 处理基础设施配置,Team Topologies 补充组织设计。这些资源解决不同问题,应选择相关部分,而不是把整张书单当作前置要求。

书籍 作用 类型
Hightower, Burns & Beda, Kubernetes: Up and Running(第 3 版) Kubernetes 对象与运行 入门
Burns, Designing Distributed Systems 云原生架构模式 入门
Ibryam & Huss, Kubernetes Patterns(第 2 版) 设计模式参考 参考
Vyas et al., Production Kubernetes 生产运维深度 深入
Davis, Cloud Native Patterns 云原生应用架构 深入
Brikman, Terraform: Up and Running(第 3 版) Infrastructure as code 入门
Skelton & Pais, Team Topologies Platform engineering 的组织基础 深入

课程、论文与当前资料

Borg 论文 与 Borg, Omega, and Kubernetes 解释集群管理需求与设计历史。Kubernetes the Hard Way 展示托管服务通常隐藏的组件,使用时应遵循它规定的环境与版本假设。KubeCon / CloudNativeCon 演讲提供具体主题的经验,应把其规模与约束同自己的问题比较。

课程/资源 平台 类型
Verma et al., “Large-scale cluster management at Google with Borg”(免费) Borg 架构论文 深入
Burns et al., “Borg, Omega, and Kubernetes”(免费) Kubernetes 谱系与设计 深入
KubeCon / CloudNativeCon talks(免费) CNCF YouTube 参考

实践、工具与当前资料

用本地 kind 或 minikube 集群检查 Pod、控制器状态、调度事件、日志与就绪状态。kubectl、k9s 与 Lens 提供不同状态视图,但任何界面都不能代替理解底层资源。可以配合 Kubernetes 的 Concepts 与 Tasks 文档,每次改变一个条件。CNCF Landscape 是分类图谱,不要求安装每一种相邻工具。

资源 平台 类型
Kubernetes documentation(免费) kubernetes.io/docs 参考
Kubernetes official tutorials(免费) kubernetes.io/docs/tutorials 实践
Certified Kubernetes Administrator curriculum(考试付费) cncf.io 辅助
Kubernetes the Hard Way(免费) github.com/kelseyhightower 实践
kind / minikube(免费) kind.sigs.k8s.io / minikube.sigs.k8s.io 实践
kubectl(免费) kubectl.docs.kubernetes.io 实践
CNCF Landscape(免费) landscape.cncf.io 参考
Engineering blogs(Spotify、Shopify、Stripe) Various 参考

陷阱

误区 为什么会误导 更好的做法
把 Kubernetes 当作单一黑箱 接纳、调度、启动、就绪与路由可以分别失败。 检查失败转换对应的状态与证据。
未建立需求就增加基础设施 对小工作负载,运维复杂度可能超过自动化收益。 按实际扩展、恢复与交付需要,比较更简单的部署。
把 YAML 数量当作进展 更多配置可能掩盖关系与不一致的默认值。 保持声明可审查,并按预期行为验证渲染后的配置。
假设共享 API 保证可移植 存储、身份、网络与托管服务依赖仍可能绑定供应方。 列出依赖,测试真正需要的迁移。
让编排决定应用边界 容易部署许多服务,不表示这种分解有用。 按应用需求与协调成本选择服务和数据边界。
认为命名空间分离足以提供安全 权限、凭据、网络策略、工作负载权限与共享内核分别产生影响。 检查实际控制,采用适合租户与工作负载模型的隔离。