English
Most data processing assumes bounded datasets: a query runs over a finite collection, produces a result, and terminates. Most of the world does not work this way. Financial transactions occur continuously. Sensor readings arrive continuously. User interactions happen continuously. Representing this continuous reality as static snapshots that are periodically queried introduces latency, discards temporal information, and misaligns the processing model with the underlying process. Stream processing takes the continuous nature of data seriously as a first-class concern rather than a special case.
An event is a discrete, immutable record of something that happened — a transaction was completed, a sensor reading was taken, a user clicked a button — with a timestamp indicating when it occurred. Event-driven architecture organizes systems around events as the primary data structure: services and analyses consume and produce events through shared event infrastructure rather than communicating through direct requests. The event log — an ordered, append-only sequence of events — is the architectural primitive that makes this possible. Events are written once to the log; multiple consumers read from it independently; historical events remain available for replay, reprocessing, and analysis alongside real-time events.
Stream processing is what happens to events in the log: the continuous computation that transforms, aggregates, and joins streams of events to produce results. These two subjects — the event architecture and the processing of events — are inseparable in practice. Stream processing at scale requires the durable, high-throughput, replay-capable infrastructure that event logs provide. Event-driven architecture without stream processing produces infrastructure that moves events but cannot reason about them continuously.
Prerequisites: Distributed systems (§4.6) — streaming systems are distributed systems with specific semantics around ordering, consistency, and fault tolerance. Databases (§4.4) — streaming databases connect stream processing to familiar SQL semantics; understanding transaction models is prerequisite to understanding streaming consistency. Data engineering (§6.3) — stream processing is increasingly integrated with batch analytics through shared storage formats.
From Complex Event Processing to the Unified Streaming Model
The intellectual history of stream processing is shorter than most systems subjects, but it contains a genuine conceptual development: the recognition that processing unbounded data requires different foundations than processing bounded data, and the construction of those foundations.
Academic work on continuous queries and complex event processing (CEP) began in the 1990s. CEP systems processed streams of events to detect patterns — “alert when three failed logins occur within five minutes from the same IP address” — using rule engines and pattern matching over sliding windows. These systems were deployed in algorithmic trading, network monitoring, and telecommunications. The academic work formalized what it meant to query a stream (Arasu, Babcock, Babu, Widom’s STREAM project at Stanford in the early 2000s; the Aurora project at Brandeis and Brown), producing the CQL (Continuous Query Language) that extended SQL with window and sequence operators.
What the early systems lacked was scale. CEP systems processed thousands of events per second; the internet-scale platforms of the late 2000s processed millions. LinkedIn, Google, and Facebook were each generating log data at rates that overwhelmed both batch processing systems (too slow) and existing stream processors (not scalable enough). The practical response at each company was the same: a distributed, high-throughput, append-only log that durably stored events and allowed consumers to read them at their own pace.
Jay Kreps, at LinkedIn, built Kafka around 2010 and published “Kafka: a Distributed Messaging System for Log Processing” at NetDB 2011. But his more significant intellectual contribution was the 2013 essay “The Log: What every software engineer should know about real-time data’s unifying abstraction.” Kreps argued that the append-only log was not merely a message queue but a foundational primitive: because logs are immutable and replayable, they enable decoupling between producers and consumers, historical reconstruction of state, and the integration of real-time and batch processing over the same data. The log is what makes event-driven architecture coherent rather than merely fashionable — it provides the durability and replay capability that give events their power as a primary data structure. Kafka, open-sourced in 2011 and donated to the Apache Software Foundation in 2012, became the dominant log implementation, and Kreps went on to found Confluent in 2014 to commercialize it.
The streaming processing semantics problem had a different origin. Building scalable stream processors was one challenge; specifying what they should compute, precisely and correctly, was another. The difficulty was time: events arrive at a stream processor in processing order (the order they are received) but they occurred in event order (the order they happened in the world). Network delays, server restarts, and distributed systems’ inherent asynchrony create gaps between these orderings. A query computing a five-minute window of purchase totals might receive events out of event-time order, with some events arriving tens of seconds or minutes late. What should the system compute, and when?
The Millwheel system at Google (Akidau et al., 2013), followed by the Dataflow model (Akidau, Bradshaw, Chambers, Cherryak, et al., VLDB 2015), provided the framework. The Dataflow model is organized around four questions: what results are being computed (transformations and aggregations), where in event time results are computed (windowing by event time), when in processing time results are materialized (triggering), and how earlier results are refined as late data arrives (accumulation and retraction). The paper introduced watermarks as the mechanism by which a streaming system estimates its completeness with respect to event time: a watermark is a lower bound on event timestamps that the system believes it has seen, providing a heuristic for when a window can be considered complete and its results emitted. Apache Beam, the programming model based on Dataflow, was open-sourced by Google in 2016 and provides a portable API that runs on multiple execution engines (Flink, Spark, Dataflow).
Apache Flink, developed at the Technical University of Berlin and open-sourced in 2014, emerged as the dominant open-source engine for stateful stream processing. Flink’s architectural commitments — event-time processing as the default, exactly-once semantics through distributed snapshots (based on Chandy-Lamport snapshot algorithm), managed state with embedded RocksDB state stores — aligned with the Dataflow model and addressed the limitations of earlier engines. Flink’s distributed snapshot mechanism, published as the Asynchronous Barrier Snapshotting (ABS) algorithm by Carbone et al. in 2017, allows consistent checkpoints of distributed state without stopping processing, enabling fault tolerance with low overhead. Flink’s influence on the semantics of production streaming has been substantial — Kafka Streams, AWS Kinesis Data Analytics, and other managed services have adopted compatible time and fault-tolerance semantics.
The current frontier is streaming databases: systems that expose streaming computation through SQL and maintain continuously updated query results that consumers can query with low latency. Materialized views — precomputed query results that are maintained as the underlying data changes — are the core abstraction. Materialize, founded by Frank McSherry (one of the designers of differential dataflow) and Nikhil Benesch, built a streaming database on the differential dataflow model that maintains SQL materialized views with strong consistency guarantees. RisingWave, Apache Flink SQL, and ksqlDB each pursue similar goals with different architectural choices. The convergence of streaming with lakehouse table formats — Apache Iceberg, Apache Hudi, Delta Lake — allows Flink and other stream processors to ingest directly into open table formats, unifying real-time and historical analysis over the same data without ETL. This convergence is dissolving the architectural boundary between the operational (streaming) and analytical (batch) tiers that had been a defining feature of data architecture for a decade.
Unbounded Data, Time, and Stateful Computation
Unbounded Data and the Windowing Problem
Traditional data processing assumes bounded datasets: a query operates on a finite collection, terminates with a complete result, then begins again. Streaming reverses this: the data is unbounded, processing is continuous, and when to produce results is itself a design decision. The implications cascade. Memory usage must be bounded even though input is not — you cannot accumulate events indefinitely. Results must be emitted before all data has arrived — you must decide when to act on incomplete information. Correctness must be defined for computations that never terminate.
Windowing divides the unbounded stream into bounded chunks for processing. Fixed windows assign each event to a window of fixed duration — hourly sales totals, daily user counts. Sliding windows overlap: a window of five minutes sliding every one minute generates a new result every minute incorporating the last five. Session windows are defined by activity: events are grouped into sessions separated by gaps of inactivity above a threshold. Each windowing scheme captures different aspects of temporal structure in the data, and choosing the right scheme requires understanding both the data and the question being asked.
The windowing decision is inseparable from the time semantics decision. Fixed windows by processing time are simple to implement — each event goes into the window current at the time it is processed — but produce results that may not correspond to any meaningful real-world interval if events arrive out of order. Fixed windows by event time compute over the interval when events actually occurred, but require the processor to wait for late-arriving events before closing a window. The tradeoff between correctness (waiting for late events) and latency (emitting results quickly) has no universal answer; the right configuration depends on the acceptable latency, the distribution of event lateness, and the cost of incorrect results.
Event Time, Processing Time, and Watermarks
The central conceptual contribution of the Dataflow model is the formal treatment of the distinction between event time and processing time and the mechanisms for handling the gap between them.
Event time is when something happened. Processing time is when the stream processor sees it. In a real distributed system, these differ: a mobile app may buffer events when offline and send them when connectivity returns; a backend service may retry a failed write and deliver an event minutes after it occurred; network congestion may delay delivery of events. The gap between event time and processing time is the skew, and it varies across events, across time, and across data sources.
A watermark is a lower bound on event times that the system believes it has not yet seen: a watermark of T means the system believes all events with event time ≤ T have already been observed. When a window’s event-time interval falls entirely before the watermark, the system infers the window is complete and emits its result. Watermarks are necessarily heuristic: if they advance too aggressively, late events arrive after the window has closed, producing incorrect results or requiring retraction. If they advance too conservatively, windows close late, introducing latency. Production watermarking algorithms estimate the distribution of event lateness from historical data and balance these costs accordingly.
Late data — events arriving after the watermark has passed their window — requires a handling policy. Options include discarding late events (simplest, incorrect when late events are significant), emitting revised results when late events arrive (correct but requires downstream systems to handle corrections), and using an allowedLateness parameter to extend how long a window accepts late events before final closure. The right policy depends on whether late events are anomalies or expected occurrences, and whether downstream consumers can handle updated results.
Exactly-Once Semantics and Stateful Processing
Many useful streaming computations require state: running counts, joins between streams, session detection, fraud pattern matching. In a distributed system where processors can fail and restart, maintaining state correctly across failures requires care. The three guarantees that stream processors provide are at-most-once (messages may be lost on failure — simplest, appropriate for loss-tolerant metrics), at-least-once (messages are redelivered on failure, potentially duplicated — requires idempotent processing), and exactly-once (each message affects state exactly once — requires coordination between the processor and its state store).
Exactly-once in a distributed stream processor is implemented through distributed snapshots. At regular intervals, the processor takes a consistent snapshot of all its distributed state — capturing the processing positions (offsets in input partitions) and the state store contents — and commits the snapshot to durable storage. On restart after failure, the processor restores from the most recent snapshot and reprocesses any input since the snapshot, which is safe because the snapshot includes the input positions. If processing produces externally visible side effects (writes to a database, Kafka output), the at-least-once reprocessing must be made idempotent or transactional to achieve end-to-end exactly-once.
Flink’s ABS algorithm takes snapshots without stopping processing by injecting barriers — special records that flow through the processing graph — and waiting for each operator to checkpoint its state when it has received barriers from all its inputs. This allows consistent distributed snapshots with minimal impact on throughput. The state store, embedded in each Flink task, uses RocksDB for large state that exceeds memory and in-memory hash maps for smaller state, providing a consistent interface across both.
Streaming joins — combining events from two streams based on matching keys — are among the most complex stateful operations. A stream-to-stream join requires buffering events from both streams until matching partners arrive, with decisions about how long to buffer (state grows indefinitely otherwise), how to handle unmatched events, and what time semantics govern the join. A stream-to-table join — enriching stream events with reference data from a slowly changing table — requires keeping the table snapshot current, deciding how to handle updates to the table that affect already-processed events, and managing the state size as the table grows.
What Studying This Changes
Stream processing changes how practitioners think about time, state, and the architecture of data-intensive systems.
The first change is recognizing when problems have streaming structure. Not all continuous data problems require full streaming infrastructure — many can be addressed with batch processing on recent data or with well-designed polling intervals. But some problems are genuinely continuous: fraud detection at transaction time, real-time dashboards, event-driven microservices, time-series analysis at scale. The practitioner who understands streaming can identify these cases early and design appropriate architectures; the practitioner who thinks only in batch terms produces systems with latency or correctness problems that are expensive to fix after the fact.
The second change is fluency with temporal semantics. Event time versus processing time is not a subtle distinction; it is the difference between computing what you intend and computing something plausible-looking but wrong. Practitioners who have internalized this distinction make it a first-class design concern: what time does this window operate on? what is the expected event lateness distribution? what is the watermark strategy? what happens to late events? These questions have specific answers that determine whether the streaming computation is correct, and they need to be answered during design rather than discovered during incident review.
The third change is operational awareness specific to continuous systems. A batch job fails visibly when it fails and succeeds when it completes. A streaming job can be subtly broken while appearing to run: consumer lag growing slowly, watermarks stuck, state stores growing unboundedly, exactly-once guarantees violated by misconfigured side effects. Practitioners who have studied streaming systems know what metrics to monitor, what anomalies to look for, and what operational knobs to adjust — knowledge that is not transferable from batch processing experience.
Resources
Books and Texts
Akidau, Chernyak, and Lax’s Streaming Systems (O’Reilly, 2018) is the canonical treatment of streaming semantics. Written by three of the principal designers of Google Cloud Dataflow and Apache Beam, it is the definitive account of the conceptual framework — event time versus processing time, windowing and triggering, watermarks, exactly-once guarantees — that underlies all serious streaming work. The prose is unusually clear and the architecture of the book (building from first principles to the unified streaming model) is exemplary. No other source provides this conceptual foundation with comparable rigor and accessibility. It is the right first book for anyone approaching streaming seriously.
Narkhede, Shapira, and Palino’s Kafka: The Definitive Guide (2nd ed., O’Reilly, 2021) covers Kafka’s architecture, consumer and producer APIs, operations, and use at the depth production work requires. Given Kafka’s dominance in event streaming infrastructure, this is effectively required reading for practitioners using streaming in production. The book covers the internals — log segments, partition replication, consumer group rebalancing — that are necessary for diagnosing production problems, not just the API surface that tutorials cover.
Stopford’s Designing Event-Driven Systems (O’Reilly, 2018, free at confluent.io) covers event-driven architectural patterns — event sourcing, CQRS, choreography versus orchestration, saga patterns for distributed transactions — and their integration with Kafka and streaming processors. It is the architectural complement to Streaming Systems’ technical depth.
Hueske and Kalavri’s Stream Processing with Apache Flink (O’Reilly, 2019) is the standard reference for engineers working with Flink specifically. It covers Flink’s DataStream and Table APIs, state management, exactly-once semantics, and deployment with the depth that production use requires.
| Book | Role | Type |
|---|---|---|
| Akidau, Chernyak & Lax, Streaming Systems | Canonical streaming semantics entry | Entry |
| Narkhede, Shapira & Palino, Kafka: The Definitive Guide (2nd ed.) | Kafka architecture and production operation | Entry |
| Stopford, Designing Event-Driven Systems (free) | Event-driven architectural patterns | Depth |
| Hueske & Kalavri, Stream Processing with Apache Flink | Flink-specific reference | Reference |
Courses, Papers, and Current Sources
The Dataflow model paper (Akidau et al., VLDB 2015, free) is the foundational paper underlying Apache Beam and the conceptual framework of Streaming Systems. Reading it after the book provides the formal precision that the book’s accessible treatment sometimes elides. Kreps’s “The Log: What every software engineer should know about real-time data’s unifying abstraction” (2013, free at engineering.linkedin.com) is the most compelling account of why event logs are a foundational primitive. Both are short (the Dataflow paper is 12 pages; the Log essay is roughly 30 minutes to read) and worth reading as primary sources.
The Apache Flink documentation and especially the section on stateful stream processing covers the checkpoint mechanism, state backends, and exactly-once semantics with the level of detail that the book supplements but does not replace. Similarly, the Apache Kafka documentation — particularly the sections on replication, consumer groups, and the offset management protocol — provides detail that the Definitive Guide summarizes at a higher level.
The Confluent blog (confluent.io/blog) is the most consistently high-quality source for contemporary streaming content, covering Kafka internals, schema evolution, streaming patterns, and production experiences at scale. The Materialize blog covers streaming databases and differential dataflow from practitioners who built the system. Engineering blogs from LinkedIn (the original Kafka team), Netflix, Uber, Spotify, and Yelp document production-scale streaming in ways that books and papers cannot.
| Resource | Platform | Type |
|---|---|---|
| Akidau et al., “The Dataflow Model” (VLDB 2015, free) | Formal foundations | Depth |
| Kreps, “The Log” essay (free) | Foundational essay on event logs | Depth |
| Apache Flink documentation (free) | Authoritative stateful streaming reference | Reference |
| Apache Kafka documentation (free) | Authoritative Kafka reference | Reference |
| Apache Beam documentation (free) | Unified programming model reference | Reference |
| Confluent blog (free) | Contemporary streaming content at scale | Reference |
| Materialize blog (free) | Streaming database and differential dataflow | Reference |
Practice, Tools, and Current Sources
Building a small streaming application is the most direct learning path. The Kafka Quickstart (kafka.apache.org) sets up a local Kafka cluster in minutes. The Flink Quickstart (flink.apache.org) provides a template for a streaming application. A productive first project: consume a stream of events from Kafka, apply a windowed aggregation in Flink (sum a metric over five-minute event-time windows with a thirty-second allowed lateness), and write results back to Kafka. This project encounters the key concepts — event time versus processing time, watermarks, window triggering, late-event handling — in a context where the behavior is observable.
Kafka’s consumer lag monitoring (kafka-consumer-groups.sh --describe) makes the key operational metric concrete: the gap between the latest offset in each partition and the current consumer offset. A growing lag indicates that the consumer is falling behind. Running a consumer, deliberately making it slow, and watching the lag grow — then fixing the performance problem and watching the lag recover — is the most effective way to internalize what consumer lag means and why it matters.
Flink’s web UI (available locally by default when running Flink) shows checkpoint history, operator state sizes, and backpressure metrics. Running a stateful Flink job locally and observing these metrics while varying the workload makes the operational characteristics of stateful streaming concrete.
| Resource | Platform | Type |
|---|---|---|
| Kafka Quickstart (free) | kafka.apache.org | Practice |
| Kafka Streams documentation (free) | kafka.apache.org/documentation/streams | Reference |
| Flink Quickstart (free) | flink.apache.org | Practice |
| Kafka consumer lag monitoring | kafka-consumer-groups.sh | Practice |
| Flink web UI | local Flink cluster | Practice |
| Apache Beam Playground (free) | beam.apache.org | Practice |
| Confluent Developer Kafka and Flink tutorials (free; cloud features may be paid) | developer.confluent.io | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Kafka as streaming | Kafka has become so dominant that practitioners often equate streaming with Kafka, treating Kafka knowledge as streaming knowledge. Kafka is the most prominent event log implementation, but streaming semantics — event time, windowing, stateful processing, exactly-once — exist largely independently of any specific technology. A practitioner who knows Kafka well but has not studied streaming semantics will build systems with wrong time semantics even using Kafka correctly. | Study streaming concepts alongside specific technologies. Streaming Systems covers the conceptual content; Kafka and Flink are implementations of broader ideas. Kafka knowledge is necessary but not sufficient for streaming competence. |
| Underestimating time handling | The distinction between event time and processing time seems subtle until a production incident reveals it is not. Systems that use processing time when event time is intended compute results that look approximately right under normal conditions and become obviously wrong when any source of latency or reordering appears — mobile apps going offline and reconnecting, backend retries, clock drift. These failures are silent in testing and visible only in production. | Allocate serious study time to time semantics before building any streaming system. Work through the canonical examples in Streaming Systems — the mobile game score example in particular — until the behavior of event-time windows under late data is second nature. Then explicitly specify the time semantics of every window in every streaming application and document why that choice is correct for the data. |
| Exactly-once confusion | “Exactly-once” appears simple but means different things in different contexts. Flink provides exactly-once state updates but not exactly-once side effects to external systems unless those systems support transactions or idempotent writes. Kafka transactions provide exactly-once semantics between Kafka topics but not to non-Kafka sinks. Engineers who adopt exactly-once guarantees without understanding what they cover build systems with subtle duplication bugs that appear only on failure paths. | For any streaming system, specify the exactly-once boundary explicitly: exactly-once with respect to what operations, between which components, under what failure scenarios. The engine documentation and architecture papers answer these questions precisely. Test failure scenarios — kill a Flink task manager mid-checkpoint, restart it, verify that exactly-once is maintained — rather than assuming the guarantee holds. |
| Schema evolution as an afterthought | Event logs are append-only and retain historical events. When the event schema changes, older events remain encoded against older schemas; newer events use newer schemas; multiple consumer versions may read from the same log simultaneously. Engineers who do not design for schema evolution from the beginning discover that their systems cannot evolve without breaking consumers or reprocessing history — an expensive problem to fix after events are in production. | Use schema registries (Confluent Schema Registry, AWS Glue Schema Registry) and evolution-aware serialization formats (Avro, Protobuf) from the beginning. Specify the backward and forward compatibility requirements for each event type before deploying. Test that older consumers can read newer events and newer consumers can read older events before any schema change reaches production. |
| Event sourcing as a default | Event sourcing — using events as the source of truth, deriving all state by replaying them — has genuine advantages (complete audit trail, temporal queries, reprocessability) and genuine costs (schema evolution complexity, storage growth, query latency for current-state reads). Some engineers adopt it as a default architectural choice because it aligns with streaming infrastructure rather than because it fits the use case. Systems built this way accumulate the costs without necessarily realizing the benefits. | Evaluate event sourcing on its specific merits for each use case. Read both the enthusiastic advocacy and the critical accounts of event-sourced systems in production before deciding. The costs compound over time; the benefits require disciplined design to realize. |
| Ignoring consumer lag and operational health | Streaming systems fail differently from batch systems. They do not crash visibly when they fall behind; they accumulate lag silently, producing increasingly stale results while appearing to run. Operators who do not monitor consumer lag, watermark progress, and checkpoint duration regularly miss the early warning signs of streaming system degradation. | Establish monitoring for consumer lag, checkpoint success rate, checkpoint duration, and operator backpressure before deploying any streaming job to production. Set alerts when consumer lag exceeds thresholds that would indicate unacceptable result latency. Treat streaming operational health as a continuous concern rather than something to investigate only after incidents. |
中文
大多数数据处理都假设数据集是有界的：一次查询在一个有限集合上运行，产生结果，然后终止。但世界上的大多数过程并不是这样运行的。金融交易持续发生。传感器读数持续到达。用户交互持续出现。把这种连续现实表示为定期查询的静态快照，会引入延迟，丢弃时间信息，并使处理模型与底层过程错位。流处理把数据的连续性当作一等关切，而不是特殊情况。
事件是一条离散、不可变的记录，记录某件已经发生的事——一笔交易完成了，一次传感器读数被采集了，一个用户点击了按钮——并带有说明其发生时间的 timestamp。事件驱动架构以事件作为主要数据结构来组织系统：服务和分析任务通过共享事件基础设施消费和生产事件，而不是通过直接请求彼此通信。事件日志——一条有序、append-only 的事件序列——是使这一点成为可能的架构原语。事件被一次写入日志；多个消费者可以独立读取；历史事件会继续保留，可用于 replay、reprocessing 和 analysis，并与实时事件一起使用。
流处理就是对日志中的事件所做的事情：一种连续计算，用来转换、聚合和 join 事件流，并产生结果。这两个主题——事件架构与事件处理——在实践中不可分离。大规模流处理需要事件日志提供的持久、高吞吐、可 replay 的基础设施。没有流处理的事件驱动架构，只会产生能够移动事件、却无法连续推理事件的基础设施。
前置知识：分布式系统（§4.6）——流式系统是带有特定排序、一致性和容错语义的分布式系统。数据库（§4.4）——流式数据库把流处理连接到熟悉的 SQL 语义；理解事务模型，是理解流式一致性的前提。数据工程（§6.3）——流处理正越来越多地通过共享存储格式与批处理分析整合。
从 Complex Event Processing 到统一流模型
流处理的智识史比大多数系统主题都短，但它包含一种真正的概念发展：认识到处理无界数据需要不同于处理有界数据的基础，并构建出这些基础。
关于 continuous queries 和 complex event processing（CEP）的学术工作始于 1990 年代。CEP 系统处理事件流来检测模式——例如“当同一 IP 地址在五分钟内出现三次登录失败时发出警报”——使用 rule engines 和 sliding windows 上的 pattern matching。这些系统被部署在算法交易、网络监控和电信领域。学术工作形式化了“查询一个 stream”到底意味着什么（Arasu、Babcock、Babu、Widom 在 2000 年代早期于 Stanford 的 STREAM 项目；Brandeis 和 Brown 的 Aurora 项目），并产生了 CQL（Continuous Query Language），它用 window 和 sequence operators 扩展了 SQL。
早期系统缺少的是规模。CEP 系统每秒处理数千事件；2000 年代后期的互联网规模平台则需要每秒处理数百万事件。LinkedIn、Google 和 Facebook 都在产生这样的日志数据速率：batch processing systems 太慢，已有 stream processors 又不够可扩展。每家公司的实践回应都相同：构建一个分布式、高吞吐、append-only log，持久存储事件，并允许消费者按自己的节奏读取。
Jay Kreps 在 LinkedIn 于 2010 年左右构建 Kafka，并在 NetDB 2011 发表 “Kafka: a Distributed Messaging System for Log Processing”。但他更重要的智识贡献，是 2013 年的文章 “The Log: What every software engineer should know about real-time data’s unifying abstraction”。Kreps 认为，append-only log 不只是 message queue，而是一种基础原语：因为 logs 是不可变且可 replay 的，它们支持 producers 与 consumers 之间的解耦、历史状态重建，以及在同一份数据上整合 real-time 和 batch processing。Log 使事件驱动架构成为一种连贯体系，而不只是流行概念——它提供了 durability 和 replay capability，使事件能够作为主要数据结构发挥力量。Kafka 于 2011 年开源，并于 2012 年捐赠给 Apache Software Foundation，随后成为主导性 log implementation；Kreps 则于 2014 年创立 Confluent，将其商业化。
流处理语义问题有不同来源。构建可扩展 stream processors 是一个挑战；精确且正确地规定它们应该计算什么，则是另一个挑战。困难在于时间：事件按照 processing order 到达 stream processor，也就是被接收的顺序；但它们按照 event order 发生，也就是现实世界中发生的顺序。网络延迟、服务器重启，以及分布式系统固有的异步性，会在这两种顺序之间制造差距。一个计算五分钟购买总额 window 的 query，可能收到 event-time order 之外的事件，其中一些事件会迟到几十秒甚至几分钟。系统应该计算什么，又应该在什么时候计算？
Google 的 Millwheel 系统（Akidau 等，2013），以及后来的 Dataflow model（Akidau、Bradshaw、Chambers、Chernyak 等，VLDB 2015）提供了框架。Dataflow model 围绕四个问题组织：计算什么结果（transformations 和 aggregations），在 event time 的哪里计算结果（按 event time windowing），在 processing time 的什么时候 materialize 结果（triggering），以及当 late data 到达时如何 refine 早先结果（accumulation 和 retraction）。这篇论文引入了 watermarks，作为流式系统估计其相对于 event time 的完整性的机制：watermark 是系统认为自己已经看到的 event timestamps 的下界，为一个 window 何时可以被认为完整、并发出结果提供启发式判断。Apache Beam 是基于 Dataflow 的 programming model，由 Google 于 2016 年开源，并提供可以运行在多个 execution engines（Flink、Spark、Dataflow）上的 portable API。
Apache Flink 由 Technical University of Berlin 开发，并于 2014 年开源，后来成为有状态流处理领域的主导开源引擎。Flink 的架构承诺——默认使用 event-time processing，通过 distributed snapshots 实现 exactly-once semantics（基于 Chandy-Lamport snapshot algorithm），并用嵌入式 RocksDB state stores 管理状态——与 Dataflow model 对齐，也处理了早期引擎的限制。Flink 的 distributed snapshot mechanism 由 Carbone 等人于 2017 年以 Asynchronous Barrier Snapshotting（ABS）algorithm 发表，它可以在不停止处理的情况下对分布式状态进行一致 checkpoint，从而以低开销实现 fault tolerance。Flink 对生产流处理语义的影响很大——Kafka Streams、AWS Kinesis Data Analytics 以及其他 managed services 都采用了兼容的时间和容错语义。
当前前沿是 streaming databases：这些系统通过 SQL 暴露流式计算，并维护持续更新的 query results，供消费者以低延迟查询。Materialized views——预先计算、并随底层数据变化而维护的查询结果——是核心抽象。Materialize 由 Frank McSherry（differential dataflow 的设计者之一）和 Nikhil Benesch 创立，它基于 differential dataflow model 构建流式数据库，以强一致性保证维护 SQL materialized views。RisingWave、Apache Flink SQL 和 ksqlDB 都以不同架构选择追求类似目标。流处理与 lakehouse table formats——Apache Iceberg、Apache Hudi、Delta Lake——的融合，使 Flink 和其他 stream processors 能直接写入开放表格式，在同一份数据上统一 real-time 和 historical analysis，而无需 ETL。这种融合正在消解 operational（streaming）层与 analytical（batch）层之间的架构边界，而这种边界曾在过去十年的数据架构中具有定义性地位。
无界数据、时间与有状态计算
无界数据与窗口问题
传统数据处理假设数据集是有界的：一次 query 在一个有限集合上运行，产生完整结果，然后开始下一次。Streaming 反过来：数据是无界的，处理是连续的，而什么时候产生结果本身就是设计决策。其影响会层层展开。即使输入无界，内存使用也必须有界——你不能无限期累积事件。结果必须在所有数据到达之前发出——你必须决定何时基于不完整信息采取行动。正确性也必须为永不终止的计算重新定义。
Windowing 会把无界 stream 切分成有界 chunks 以供处理。Fixed windows 会把每个事件分配到固定时长窗口中——例如每小时销售总额、每日用户数。Sliding windows 会重叠：一个五分钟 window 每一分钟 sliding 一次，就会每分钟产生一个新结果，包含过去五分钟数据。Session windows 由活动定义：事件被分组为 sessions，而 session 之间由超过某个阈值的不活跃间隔分隔。每种 windowing scheme 都捕捉数据中不同的时间结构；选择正确方案，需要理解数据本身和所提出的问题。
Windowing 决策无法与时间语义决策分开。按 processing time 设置 fixed windows 很容易实现——每个事件进入其被处理时当前的 window——但如果事件乱序到达，结果可能不对应任何有意义的现实时间区间。按 event time 设置 fixed windows，会在事件实际发生的时间区间上计算，但要求 processor 在关闭 window 之前等待 late-arriving events。正确性（等待迟到事件）与延迟（快速发出结果）之间的权衡，没有通用答案；正确配置取决于可接受延迟、事件迟到分布，以及错误结果的成本。
Event Time、Processing Time 与 Watermarks
Dataflow model 的中心概念贡献，是形式化处理 event time 与 processing time 的区别，并提供处理二者差距的机制。
Event time 是事情发生的时间。Processing time 是 stream processor 看到它的时间。在真实分布式系统中，二者不同：移动应用可能在离线时缓存事件，并在连接恢复后发送；后端服务可能重试一次失败写入，并在事件发生数分钟后才投递；网络拥塞也可能延迟事件投递。Event time 与 processing time 之间的差距就是 skew，而且它会随事件、时间和数据源变化。
Watermark 是系统认为自己尚未看到的 event times 的下界：一个值为 T 的 watermark，表示系统认为所有 event time ≤ T 的事件都已经被观察到。当一个 window 的 event-time interval 完全早于 watermark 时，系统推断这个 window 已经完整，并发出结果。Watermarks 必然是启发式的：如果推进得太激进，late events 会在 window 关闭后到达，产生错误结果或要求 retraction；如果推进得太保守，windows 会很晚才关闭，引入延迟。生产 watermarking algorithms 会根据历史数据估计事件迟到分布，并据此平衡这些成本。
Late data——在 watermark 已经越过其 window 后才到达的事件——需要处理策略。可选方案包括丢弃 late events（最简单，但当 late events 重要时不正确）、在 late events 到达时发出修订结果（正确，但要求下游系统能处理 corrections），以及使用 allowedLateness 参数，延长 window 在最终关闭前接受 late events 的时间。正确策略取决于 late events 是异常还是预期情况，也取决于 downstream consumers 是否能处理更新结果。
Exactly-Once Semantics 与有状态处理
许多有用的流式计算都需要状态：running counts、streams 之间的 joins、session detection、fraud pattern matching。在 processors 可能失败并重启的分布式系统中，跨失败正确维护状态需要小心。Stream processors 提供的三类保证是 at-most-once（失败时 messages 可能丢失——最简单，适合 loss-tolerant metrics）、at-least-once（失败时 messages 会被重新投递，可能重复——要求 idempotent processing），以及 exactly-once（每条 message 只影响状态一次——要求 processor 与其 state store 之间协调）。
分布式 stream processor 中的 exactly-once 通过 distributed snapshots 实现。系统会定期对所有分布式状态做一致 snapshot——捕获 processing positions（input partitions 中的 offsets）和 state store contents——并把 snapshot 提交到持久存储。失败后重启时，processor 从最近 snapshot 恢复，并重新处理 snapshot 之后的输入；这是安全的，因为 snapshot 包含输入位置。如果 processing 产生外部可见 side effects（写入数据库、Kafka output），那么 at-least-once reprocessing 必须通过 idempotent 或 transactional 方式处理，才能实现 end-to-end exactly-once。
Flink 的 ABS algorithm 通过注入 barriers——流经 processing graph 的特殊 records——在不停止处理的情况下完成 snapshots，并等待每个 operator 在从所有 inputs 接收到 barriers 后 checkpoint 自己的状态。这允许以对吞吐量影响很小的方式获得一致 distributed snapshots。嵌入在每个 Flink task 中的 state store，对超过内存的大状态使用 RocksDB，对较小状态使用 in-memory hash maps，并在二者之间提供一致接口。
Streaming joins——基于 matching keys 组合两个 streams 中的事件——是最复杂的有状态操作之一。Stream-to-stream join 需要 buffering 两个 streams 中的事件，直到匹配对象到达；这涉及要 buffer 多久（否则 state 会无限增长）、如何处理 unmatched events，以及 join 使用什么时间语义。Stream-to-table join——用来自缓慢变化 table 的 reference data enrich stream events——需要保持 table snapshot 更新，决定如何处理那些会影响已处理事件的 table 更新，并随着 table 增长管理 state size。
学习这一部分会改变什么
流处理会改变实践者思考时间、状态，以及数据密集型系统架构的方式。
第一个变化，是识别哪些问题具有 streaming structure。并非所有连续数据问题都需要完整流式基础设施——许多问题可以通过对近期数据进行 batch processing，或通过设计良好的 polling intervals 处理。但某些问题确实是连续的：交易发生时的 fraud detection、real-time dashboards、event-driven microservices、大规模 time-series analysis。理解流处理的实践者能早期识别这些情况，并设计合适架构；只按 batch terms 思考的实践者，则会做出带有延迟或正确性问题的系统，而这些问题事后修复代价很高。
第二个变化，是熟练处理时间语义。Event time 与 processing time 不是微妙小区别；它是“计算你真正想计算的东西”与“计算一个看起来合理但错误的东西”之间的区别。内化这种区别的实践者会把它作为一等设计关切：这个 window 基于什么时间运行？预期 event lateness distribution 是什么？watermark strategy 是什么？late events 会怎样处理？这些问题有具体答案，并决定流式计算是否正确；它们需要在设计阶段回答，而不是在事故复盘中发现。
第三个变化，是针对连续系统的运维意识。Batch job 失败时会明显失败，完成时也会明显成功。Streaming job 则可能在看似运行的情况下微妙损坏：consumer lag 缓慢增长，watermarks 卡住，state stores 无界增长，exactly-once guarantees 因 side effects 配置错误而被破坏。学习过 streaming systems 的实践者知道该监控哪些 metrics、寻找哪些 anomalies，以及调整哪些 operational knobs——这些知识不能从 batch processing 经验中直接迁移。
资源
书籍与文本
Akidau、Chernyak 和 Lax 的 Streaming Systems（O’Reilly，2018）是流处理语义的经典处理。三位作者都是 Google Cloud Dataflow 和 Apache Beam 的主要设计者，这本书是对概念框架的权威说明——event time versus processing time、windowing and triggering、watermarks、exactly-once guarantees——这些概念支撑所有严肃流处理工作。其行文异常清晰，书的架构也堪称典范：从第一原则逐步构建到统一流模型。没有其他来源能以同等严谨性和可读性提供这种概念基础。对任何严肃接触流处理的人来说，这是正确的第一本书。
Narkhede、Shapira 和 Palino 的 Kafka: The Definitive Guide（第 2 版，O’Reilly，2021）以生产工作所需深度覆盖 Kafka 架构、consumer 和 producer APIs、运维以及使用方式。鉴于 Kafka 在事件流基础设施中的主导地位，这本书实际上是生产中使用流处理的实践者必读。它覆盖的内部机制——log segments、partition replication、consumer group rebalancing——正是诊断生产问题所必需的，而不只是教程覆盖的 API 表面。
Stopford 的 Designing Event-Driven Systems（O’Reilly，2018，可在 confluent.io 免费获取）覆盖事件驱动架构模式——event sourcing、CQRS、choreography versus orchestration、用于 distributed transactions 的 saga patterns——以及它们与 Kafka 和 stream processors 的整合。它是 Streaming Systems 技术深度的架构补充。
Hueske 和 Kalavri 的 Stream Processing with Apache Flink（O’Reilly，2019）是使用 Flink 的工程师的标准参考。它覆盖 Flink 的 DataStream 和 Table APIs、state management、exactly-once semantics，以及生产使用所需深度的部署问题。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Akidau, Chernyak & Lax, Streaming Systems | 经典流处理语义入口 | 入门 |
| Narkhede, Shapira & Palino, Kafka: The Definitive Guide（第 2 版） | Kafka 架构与生产运维 | 入门 |
| Stopford, Designing Event-Driven Systems（免费） | 事件驱动架构模式 | 深入 |
| Hueske & Kalavri, Stream Processing with Apache Flink | Flink 专门参考 | 参考 |
课程、论文与当前资料
Dataflow model paper（Akidau 等，VLDB 2015，免费）是 Apache Beam 和 Streaming Systems 概念框架背后的基础论文。读完书后阅读它，可以获得书中易读处理有时略去的形式化精确性。Kreps 的 “The Log: What every software engineer should know about real-time data’s unifying abstraction”（2013，可在 engineering.linkedin.com 免费获取）是关于 event logs 为什么是基础原语的最有说服力说明。二者都很短（Dataflow 论文 12 页；Log 文章约 30 分钟可读完），都值得作为一手材料阅读。
Apache Flink documentation，尤其是 stateful stream processing 部分，覆盖 checkpoint mechanism、state backends 和 exactly-once semantics，细节层级足以补充但不能被书取代。类似地，Apache Kafka documentation——特别是 replication、consumer groups 和 offset management protocol 部分——提供了 Definitive Guide 在更高层概括的细节。
Confluent blog（confluent.io/blog）是当代流处理内容中持续质量最高的来源，覆盖 Kafka internals、schema evolution、streaming patterns，以及大规模生产经验。Materialize blog 则从系统构建者视角覆盖 streaming databases 和 differential dataflow。LinkedIn（原始 Kafka 团队）、Netflix、Uber、Spotify 和 Yelp 的工程博客记录了生产规模流处理经验，这类内容是书和论文无法提供的。
| 资源 | 平台 | 类型 |
|---|---|---|
| Akidau et al., “The Dataflow Model”（VLDB 2015，免费） | 形式化基础 | 深入 |
| Kreps, “The Log” essay（免费） | 关于 event logs 的奠基文章 | 深入 |
| Apache Flink documentation（免费） | 权威有状态流处理参考 | 参考 |
| Apache Kafka documentation（免费） | 权威 Kafka 参考 | 参考 |
| Apache Beam documentation（免费） | 统一 programming model 参考 | 参考 |
| Confluent blog（免费） | 当代大规模流处理内容 | 参考 |
| Materialize blog（免费） | Streaming database 与 differential dataflow | 参考 |
实践、工具与当前资料
构建一个小型流处理应用，是最直接的学习路径。Kafka Quickstart（kafka.apache.org）可以在几分钟内建立本地 Kafka cluster。Flink Quickstart（flink.apache.org）提供 streaming application 模板。一个有效的第一个项目是：从 Kafka 消费一条事件流，在 Flink 中应用 windowed aggregation（以五分钟 event-time windows 汇总某个 metric，并设置 thirty-second allowed lateness），再把结果写回 Kafka。这个项目会在行为可观察的语境中触及关键概念——event time versus processing time、watermarks、window triggering、late-event handling。
Kafka 的 consumer lag monitoring（kafka-consumer-groups.sh --describe）会使关键运维指标变得具体：每个 partition 中最新 offset 与当前 consumer offset 之间的差距。lag 增长说明 consumer 正在落后。运行一个 consumer，故意让它变慢，观察 lag 增长；然后修复性能问题，观察 lag 恢复——这是内化 consumer lag 意义及其重要性的最有效方式。
Flink 的 web UI（本地运行 Flink 时默认可用）显示 checkpoint history、operator state sizes 和 backpressure metrics。本地运行一个有状态 Flink job，并在改变 workload 时观察这些 metrics，会使有状态流处理的运维特征具体化。
| 资源 | 平台 | 类型 |
|---|---|---|
| Kafka Quickstart（免费） | kafka.apache.org | 实践 |
| Kafka Streams documentation（免费） | kafka.apache.org/documentation/streams | 参考 |
| Flink Quickstart（免费） | flink.apache.org | 实践 |
| Kafka consumer lag monitoring | kafka-consumer-groups.sh | 实践 |
| Flink web UI | local Flink cluster | 实践 |
| Apache Beam Playground（免费） | beam.apache.org | 实践 |
| Confluent Developer Kafka and Flink tutorials（免费；cloud features 可能付费） | developer.confluent.io | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 把 Kafka 等同于 streaming | Kafka 已经如此主导，以至于实践者经常把 streaming 等同于 Kafka，把 Kafka 知识当作 streaming 知识。Kafka 是最重要的 event log implementation，但 streaming semantics——event time、windowing、stateful processing、exactly-once——在很大程度上独立于任何具体技术。一个很懂 Kafka、但没有学习 streaming semantics 的实践者，即使正确使用 Kafka，也会构建出时间语义错误的系统。 | 把流处理概念与具体技术一起学习。Streaming Systems 覆盖概念内容；Kafka 和 Flink 是更广泛思想的实现。Kafka 知识必要，但不足以构成 streaming competence。 |
| 低估时间处理 | Event time 与 processing time 的区别看似微妙，直到生产事故证明它并不微妙。当意图使用 event time，却实际使用 processing time 时，系统在正常条件下会计算出看起来大致正确的结果；但一旦出现任何延迟或重排序来源——移动应用离线后重连、后端重试、时钟漂移——结果就会明显错误。这些失败在测试中是沉默的，只有在生产中才可见。 | 在构建任何流式系统前，给时间语义分配严肃学习时间。完成 Streaming Systems 中的经典例子——尤其是 mobile game score example——直到 late data 下 event-time windows 的行为成为第二本能。然后显式规定每个 streaming application 中每个 window 的时间语义，并记录为什么这种选择对数据是正确的。 |
| Exactly-once 混乱 | “Exactly-once” 看起来简单，但在不同语境中含义不同。Flink 提供 exactly-once state updates，但除非外部系统支持 transactions 或 idempotent writes，否则它不提供对外部系统 side effects 的 exactly-once。Kafka transactions 提供 Kafka topics 之间的 exactly-once semantics，但不提供到 non-Kafka sinks 的 exactly-once。工程师如果采用 exactly-once guarantees，却不理解其覆盖范围，就会构建出只在 failure paths 上出现的微妙重复 bug。 | 对任何流式系统，都要明确规定 exactly-once boundary：相对于哪些 operations、在哪些 components 之间、在什么 failure scenarios 下 exactly-once。Engine documentation 和架构论文会精确回答这些问题。测试失败场景——在 checkpoint 中途 kill 一个 Flink task manager，重启它，验证 exactly-once 是否保持——而不是假定保证自然成立。 |
| 把 schema evolution 当成事后问题 | Event logs 是 append-only 的，并会保留历史事件。当 event schema 改变时，旧事件仍然按旧 schema 编码；新事件使用新 schema；多个 consumer versions 可能同时读取同一个 log。没有从一开始为 schema evolution 设计的工程师，会发现系统无法在不破坏 consumers 或 reprocess history 的情况下演化——而一旦事件进入生产，这就是一个昂贵问题。 | 从一开始就使用 schema registries（Confluent Schema Registry、AWS Glue Schema Registry）和 evolution-aware serialization formats（Avro、Protobuf）。在部署之前规定每种 event type 的 backward 和 forward compatibility requirements。任何 schema change 进入生产前，都要测试旧 consumers 是否能读取新 events，新 consumers 是否能读取旧 events。 |
| 把 event sourcing 当成默认选择 | Event sourcing——把 events 作为事实来源，并通过 replay 推导所有状态——确实有优势（完整 audit trail、temporal queries、reprocessability），也确实有成本（schema evolution complexity、storage growth、current-state reads 的 query latency）。有些工程师把它当作默认架构选择，不是因为它适合 use case，而是因为它与 streaming infrastructure 对齐。这样构建出的系统会积累成本，却未必真正获得收益。 | 针对每个 use case，按 event sourcing 的具体优劣评估它。决定前，既阅读支持者的论述，也阅读生产中 event-sourced systems 的批判性经验。成本会随时间复利增长；收益则需要严格设计才能实现。 |
| 忽视 consumer lag 与运维健康 | Streaming systems 的失败方式不同于 batch systems。它们落后时不会明显崩溃；它们会静默积累 lag，产生越来越 stale 的结果，同时看起来仍在运行。不定期监控 consumer lag、watermark progress 和 checkpoint duration 的 operators，会错过 streaming system degradation 的早期信号。 | 在任何 streaming job 部署到生产前，建立对 consumer lag、checkpoint success rate、checkpoint duration 和 operator backpressure 的监控。当 consumer lag 超过代表结果延迟不可接受的阈值时设置 alerts。把 streaming operational health 当成持续关切，而不是事故之后才调查的东西。 |