English
Stream processing computes over data that keeps arriving. A shop may need an updated sales total every minute while purchases continue indefinitely. A batch job computes over a bounded input; a streaming job must also decide when to publish partial results, how to revise them, and how much state to retain. Batch processing remains suitable when its delay meets the requirement, and it can preserve event timestamps perfectly well.
An event records an occurrence, such as an order being accepted. A command requests an action; an event reports something that has happened. Event-driven architecture connects producers and consumers through notifications or messages, often using queues or durable logs. A partitioned log supplies an order within each partition and independent consumer positions. Retention and compaction policies determine what remains replayable; an append-only interface does not promise that all history is kept forever.
Stream processing and event-driven architecture overlap but are not inseparable. A service can react to an event without running a continuous analytical query, and a stream processor can consume sources other than a durable log. Event sourcing, which reconstructs application state from recorded events, and CQRS, which separates command and query responsibilities, are further design choices rather than automatic consequences of using a broker.
Prerequisites: Distributed systems (§4.6) and databases (§4.4). Data engineering (§6.3) provides further context for combining continuous and batch computation.
From Complex Event Processing to the Unified Streaming Model
Continuous queries and event detection arose from applications that needed to act before a dataset was complete: monitoring, trading, telecommunications, and operational control. Later systems combined this requirement with distributed storage and recovery. The main conceptual development was specifying both the result and when that result was ready to use.
Continuous-query projects such as Stanford’s STREAM and Aurora studied computations over arriving data, while complex event processing studied patterns such as three failed logins within five minutes. STREAM’s CQL connected streams to relations: a window selects a time-varying relation from a stream, relational operators compute over it, and output operators describe changes as streams. This connection explains how familiar database operations can be reused without pretending that an unbounded input is already complete.
Large services added requirements for throughput, independent consumers, and recovery after processing failures. Durable partitioned logs separated accepting data from processing it: one consumer could fall behind or replay input without forcing every other consumer to restart. Partitioning allowed parallel consumption, while key assignment determined which related events shared an order. Retention determined how long a lagging consumer could recover from the log.
Kafka was developed at LinkedIn, described by Kreps, Narkhede, and Rao in 2011, and developed further as an Apache project. Kreps’s 2013 essay on the log explained how retaining an ordered record could support replay, independent consumption, and state reconstruction. Those benefits depend on the record actually retained and on application semantics. Replaying a log through changed code or a changing external lookup need not reproduce an earlier result.
The streaming processing semantics problem had a different origin. Building scalable stream processors was one challenge; specifying what they should compute, precisely and correctly, was another. The difficulty was time: events arrive at a stream processor in processing order (the order they are received) but they occurred in event order (the order they happened in the world). Network delays, server restarts, and distributed systems’ inherent asynchrony create gaps between these orderings. A query computing a five-minute window of purchase totals might receive events out of event-time order, with some events arriving tens of seconds or minutes late. What should the system compute, and when?
Google’s MillWheel (2013) and the Dataflow model (2015) helped organize event-time computation. The Dataflow model separates what to compute, where to group records in event time, when to emit results, and how later input revises previous output. Watermarks express progress in event time, while triggers govern emission. Apache Beam subsequently exposed a related programming model across multiple execution engines; portability still requires checking the features and semantics supported by a chosen runner.
Flink developed from the Stratosphere research project and became an Apache project in 2014. It combines distributed computation with managed state, event-time mechanisms, and recovery from checkpoints. Its checkpoint barriers coordinate a consistent recovery point across operators. Alignment can delay some inputs, and asynchronous snapshot writing does not mean checkpointing has no processing or storage cost. The contribution is a specified recovery mechanism, not a universal promise that every connected application executes effects exactly once.
Streaming databases connect these ideas to incremental view maintenance: a query result is updated as underlying records change. Materialize, RisingWave, and Flink SQL illustrate different designs in this area. Open table formats can also connect streaming ingestion to historical analysis. Sharing a storage format does not remove schema changes, data-quality work, transaction boundaries, or the need to decide how fresh a query result must be.
Unbounded Data, Time, and Stateful Computation
Define the query, its time basis, and its update policy before choosing infrastructure. A continuously published result needs a meaning at each publication, not merely a process that never exits.
Unbounded Data and the Windowing Problem
Traditional data processing assumes bounded datasets: a query operates on a finite collection, terminates with a complete result, then begins again. Streaming reverses this: the data is unbounded, processing is continuous, and when to produce results is itself a design decision. The implications cascade. Memory usage must be bounded even though input is not — you cannot accumulate events indefinitely. Results must be emitted before all data has arrived — you must decide when to act on incomplete information. Correctness must be defined for computations that never terminate.
Windowing divides the unbounded stream into bounded chunks for processing. Fixed windows assign each event to a window of fixed duration — hourly sales totals, daily user counts. Sliding windows overlap: a window of five minutes sliding every one minute generates a new result every minute incorporating the last five. Session windows are defined by activity: events are grouped into sessions separated by gaps of inactivity above a threshold. Each windowing scheme captures different aspects of temporal structure in the data, and choosing the right scheme requires understanding both the data and the question being asked.
Choose time semantics according to the question. Processing-time windows describe work observed by the processor during an interval; event-time windows group records by their assigned occurrence timestamps. An event-time system can emit early results and revise them, so the decision is not simply to wait forever for correctness or publish an incorrect answer. Specify which lateness is accepted, when results become final, and how downstream consumers handle updates.
A finite window duration does not by itself bound total state: the number of keys, retained windows, buffered matches, and delayed inputs also matter. A running sum needs little state per key, while an exact set of every distinct identifier can keep growing. Decide which data can be discarded and what information that discarding loses.
Event Time, Processing Time, and Watermarks
Event time is a timestamp assigned to represent when an event occurred; processing time is when an operator processes it. Event timestamps can themselves be inaccurate, so define their source and trust assumptions. A purchase stamped 09:59 that arrives at 10:02 belongs to the 09:00–10:00 event-time window, but to the 10:00–11:00 processing-time window. Both computations can be well-defined; they answer different questions.
A watermark expresses progress: under the source’s policy, events at or before a timestamp are expected to have arrived. Some sources can provide a strict completeness bound; others estimate it from tolerated disorder. An estimate can be violated by a late event. Window assignment, output triggers, allowed lateness, and state cleanup are separate decisions, as illustrated by Flink’s time model.
An early result may be useful before the watermark passes the window. When the 09:59 purchase arrives, a system can revise the earlier total if it still accepts late data, send the event for separate reconciliation, or exclude it under an explicit contract. Updating a result requires consumers to distinguish replacement totals from additional increments; otherwise a correction can be counted twice.
For operators with several inputs, progress is often constrained by the slowest active input. An idle partition can therefore delay results even when other partitions are busy. An idleness policy can allow progress, but must also define what happens when that partition resumes with old events. Observe both data lag and watermark progress rather than interpreting a running job as a current result.
Exactly-Once Semantics and Stateful Processing
State connects successive events: a count remembers earlier increments, a join remembers unmatched records, and a detector remembers earlier steps of a pattern. Failure recovery must reconnect input position with that state. At-most-once processing can omit work; at-least-once processing can repeat it. Exactly-once semantics describes a specified effect, usually on managed state or transactional output, rather than saying that code physically executes only once.
Suppose a checkpoint records count 40 and input position 100. The next record raises the count to 41, then the processor fails before the next checkpoint. Restoring both count 40 and position 100 and replaying that record yields 41 again. Restoring count 41 while replaying the same increment would incorrectly yield 42. This is why state and input positions need one consistent recovery point. Flink’s state model describes checkpoint and replay coordination.
External effects need their own integration. Replaying a record that already sent an email or charged a card can repeat the effect even if internal state is correct. A transactional sink, or durable duplicate detection atomically connected to the business action, can establish a stronger guarantee within its stated boundary. A broker transaction does not automatically include an arbitrary external API.
Checkpoint barriers identify recovery boundaries across a processing graph; snapshot writing and alignment have costs. A source must support the required replay, and state and connector configuration must support the claimed guarantee. Nondeterministic computations and changing external lookups also need attention if recovery must reproduce the same result.
A stream join must specify keys and time as well as retention. Joining an order with payment events may require retaining unmatched records for a defined interval. Enriching an old order with a customer’s address also requires a choice: the address now, or the address valid when the order was placed? These are query semantics, not merely storage tuning.
Backpressure slows upstream production or delivery when downstream capacity is insufficient. Buffers absorb bursts, not a permanently higher arrival rate than processing rate. Track queue growth, processing capacity, and recovery time after an interruption; replaying a backlog requires capacity beyond the continuing arrival rate if the system is to catch up.
What Studying This Changes
Choose streaming when its response time or incremental computation meets a concrete need. A periodic batch query may be easier to operate when minutes of delay are acceptable. For either design, specify the result’s time basis, freshness, and treatment of corrections.
Practice with a small sales stream containing out-of-order and duplicate records. State the expected hourly total, then fail the processor before and after a checkpoint. Check internal state and externally visible output separately. This exercise connects time semantics, replay, and business correctness more directly than comparing engine feature lists.
Resources
Books and Texts
Streaming Systems explains time, windows, triggers, and evolving results. Kafka: The Definitive Guide covers a particular log platform; Designing Event-Driven Systems connects messaging to application architecture. Stream Processing with Apache Flink develops processor-specific mechanisms. Use the conceptual material across systems, but check API, connector, and recovery details against the version you run.
| Book | Role | Type |
|---|---|---|
| Akidau, Chernyak & Lax, Streaming Systems | Time and state in stream processing | Entry |
| Shapira, Palino, Sivaram & Petty, Kafka: The Definitive Guide (2nd ed.) | Kafka architecture and production operation | Entry |
| Stopford, Designing Event-Driven Systems (free) | Event-driven architectural patterns | Depth |
| Hueske & Kalavri, Stream Processing with Apache Flink | Flink-specific reference | Reference |
Courses, Papers, and Current Sources
The Dataflow model paper separates computation from windowing, triggering, and result updates; Kreps’s The Log essay explains replay and independent consumption. Flink, Kafka, and Beam documentation specifies the available mechanisms. Confluent, Materialize, and other engineering publications offer implementation case studies. Distinguish a documented guarantee from a particular deployment experience or product comparison.
| Resource | Platform | Type |
|---|---|---|
| Akidau et al., “The Dataflow Model” (VLDB 2015, free) | Formal foundations | Depth |
| Kreps, “The Log” essay (free) | Foundational essay on event logs | Depth |
| Apache Flink documentation (free) | Authoritative stateful streaming reference | Reference |
| Apache Kafka documentation (free) | Authoritative Kafka reference | Reference |
| Apache Beam documentation (free) | Unified programming model reference | Reference |
| Confluent blog (free) | Contemporary streaming content at scale | Reference |
| Materialize blog (free) | Streaming database and differential dataflow | Reference |
Practice, Tools, and Current Sources
Build a five-minute event-time aggregation with an explicit lateness policy. Send records in order, out of order, and after state cleanup; verify the expected outputs. Then interrupt processing and check recovery. Kafka consumer offsets and Flink’s UI help observe progress, checkpoint behavior, and backpressure. Offset lag counts position distance rather than directly measuring elapsed delay, so also measure result freshness.
| Resource | Platform | Type |
|---|---|---|
| Kafka Quickstart (free) | kafka.apache.org | Practice |
| Kafka Streams documentation (free) | kafka.apache.org/documentation/streams | Reference |
| Flink Quickstart (free) | flink.apache.org | Practice |
| Kafka consumer lag monitoring | kafka-consumer-groups.sh | Practice |
| Flink web UI | local Flink cluster | Practice |
| Apache Beam Playground (free) | beam.apache.org | Practice |
| Confluent Developer Kafka and Flink tutorials (free; cloud features may be paid) | developer.confluent.io | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Equating Kafka with stream processing | A log provides storage and consumption mechanisms; queries still need time and state semantics. | Define the computation independently, then map it to broker and processor capabilities. |
| Leaving the meaning of time implicit | Event time and processing time can put the same event in different windows. | Test delayed and reordered events against a stated expected result. |
| Assuming exactly-once covers every effect | State recovery does not automatically undo or deduplicate an external action. | Specify the guarantee boundary and test both internal state and external output after failure. |
| Ignoring schema evolution | Retained events and concurrent consumer versions may use different schemas. | State compatibility requirements and test representative old and new records. |
| Adopting event sourcing without a need | Replay and audit benefits require retention, versioning, and controlled effects. | Evaluate those costs against the required history and recovery behavior. |
| Treating a running job as a current result | Lag, stuck watermarks, and growing state can hide behind a live process. | Track freshness, backlog, checkpoint health, and available catch-up capacity. |
中文
流处理对持续到达的数据进行计算。例如，商店希望每分钟更新销售总额，而购买行为持续发生，没有预定终点。批处理针对有界输入计算；流处理还需要决定何时发布部分结果、怎样修订结果，以及保留多少状态。当批处理的延迟满足需求时，它仍然适用，也完全可以保留事件时间戳。
事件记录一次发生的事情，例如订单被接受。命令请求执行动作，事件报告已经发生的事情。事件驱动架构通过通知或消息连接生产者与消费者，常使用队列或持久日志。分区日志提供各分区内部的顺序，以及独立的消费者位置。保留与压缩策略决定哪些内容仍可重放；只追加接口不保证永久保存全部历史。
流处理与事件驱动架构相互重叠，但并非不可分割。服务可以响应事件而不运行连续分析查询，流处理器也可以消费持久日志之外的数据源。事件溯源从所记录事件重建应用状态，CQRS 则分离命令与查询职责；它们都是进一步的设计选择，并非使用消息代理的自动结果。
前置知识：分布式系统（§4.6）与数据库（§4.4）。数据工程（§6.3）进一步介绍连续计算与批处理的结合。
从 Complex Event Processing 到统一流模型
连续查询与事件检测来自那些不能等数据集完整后再行动的应用：监控、交易、电信与运行控制。后来的系统把这一要求与分布式存储、恢复结合起来。主要的概念进展，是同时规定计算结果，以及何时可以使用这个结果。
Stanford 的 STREAM、Aurora 等连续查询项目研究对到达数据的计算，复杂事件处理则研究五分钟内三次登录失败等模式。STREAM 的 CQL 把流与关系连接起来：窗口从流中选出随时间变化的关系，关系算子在其上计算，输出算子把变化描述为流。这种连接解释了怎样复用熟悉的数据库操作，同时不假装无界输入已经完整。
大型服务增加了吞吐量、独立消费者与处理失败后恢复的要求。持久分区日志把接收数据与处理数据分开：一个消费者可以落后或重放输入，而不必迫使其他消费者全部重启。分区允许并行消费，键的分配决定哪些相关事件共享顺序，保留策略则决定落后的消费者能在多久之后仍从日志恢复。
Kafka 在 LinkedIn 开发，由 Kreps、Narkhede 与 Rao 在 2011 年的论文中介绍，随后作为 Apache 项目继续发展。Kreps 在 2013 年关于日志的文章，解释了保留有序记录怎样支持重放、独立消费与状态重建。这些收益依赖实际保留的记录和应用语义。用修改过的代码，或查询不断变化的外部数据来重放日志，不一定重现原来的结果。
流处理语义问题有不同来源。构建可扩展 stream processors 是一个挑战；精确且正确地规定它们应该计算什么，则是另一个挑战。困难在于时间：事件按照 processing order 到达 stream processor，也就是被接收的顺序；但它们按照 event order 发生，也就是现实世界中发生的顺序。网络延迟、服务器重启，以及分布式系统固有的异步性，会在这两种顺序之间制造差距。一个计算五分钟购买总额 window 的 query，可能收到 event-time order 之外的事件，其中一些事件会迟到几十秒甚至几分钟。系统应该计算什么，又应该在什么时候计算？
Google 的 MillWheel（2013）与 Dataflow 模型（2015）帮助组织事件时间计算。Dataflow 模型区分计算什么、在事件时间中怎样分组、何时输出结果，以及后续输入怎样修订已有输出。水位线表达事件时间的进展，触发器控制结果输出。Apache Beam 随后把相关编程模型提供给多种执行引擎；可移植性仍要求检查选定 runner 支持的功能与语义。
Flink 从 Stratosphere 研究项目发展而来，在 2014 年成为 Apache 项目。它把分布式计算与托管状态、事件时间机制、检查点恢复结合起来。检查点屏障在算子之间协调一致的恢复位置。屏障对齐可以延迟部分输入，异步写入快照也不表示检查点没有处理或存储成本。其贡献是明确的恢复机制，而非承诺所有接入应用都恰好执行一次外部效果。
流式数据库把这些思想连接到增量视图维护：底层记录变化时，查询结果随之更新。Materialize、RisingWave 与 Flink SQL 展示了这一方向的不同设计。开放表格式也可以把流式摄取连接到历史分析。但共享存储格式不会消除模式变更、数据质量工作、事务边界，或决定查询结果需要多新鲜的责任。
无界数据、时间与有状态计算
选择基础设施之前，先定义查询、时间依据与更新策略。连续发布的结果需要在每次发布时具有明确含义，而不只是让进程永不退出。
无界数据与窗口问题
传统数据处理假设数据集是有界的：一次 query 在一个有限集合上运行，产生完整结果，然后开始下一次。Streaming 反过来：数据是无界的，处理是连续的，而什么时候产生结果本身就是设计决策。其影响会层层展开。即使输入无界，内存使用也必须有界——你不能无限期累积事件。结果必须在所有数据到达之前发出——你必须决定何时基于不完整信息采取行动。正确性也必须为永不终止的计算重新定义。
Windowing 会把无界 stream 切分成有界 chunks 以供处理。Fixed windows 会把每个事件分配到固定时长窗口中——例如每小时销售总额、每日用户数。Sliding windows 会重叠：一个五分钟 window 每一分钟 sliding 一次，就会每分钟产生一个新结果，包含过去五分钟数据。Session windows 由活动定义：事件被分组为 sessions，而 session 之间由超过某个阈值的不活跃间隔分隔。每种 windowing scheme 都捕捉数据中不同的时间结构；选择正确方案，需要理解数据本身和所提出的问题。
按所要回答的问题选择时间语义。处理时间窗口描述处理器在一个区间内观察到的工作，事件时间窗口按记录所赋予的发生时间戳分组。事件时间系统可以提前输出并修订结果，因此选择并非只有为了正确无限等待，或立即发布错误答案。应规定接受怎样的迟到、结果何时最终确定，以及下游怎样处理更新。
有限的窗口时长本身不能限制总状态量：键的数量、保留的窗口、缓冲的匹配记录与延迟输入也很重要。每个键的累计总和需要较少状态，而精确保留每个不同标识的集合可以持续增长。需要决定哪些数据可以丢弃，以及丢弃会失去什么信息。
Event Time、Processing Time 与 Watermarks
事件时间是用于表示事件发生时刻的时间戳，处理时间是算子处理该事件的时刻。事件时间戳本身也可能不准确，因此需要定义其来源与信任假设。一笔标记为 09:59、在 10:02 才到达的购买，属于 09:00–10:00 的事件时间窗口，却属于 10:00–11:00 的处理时间窗口。两种计算都可以有明确定义，但回答不同问题。
水位线表达进展：按数据源的策略，某个时间戳及之前的事件预计已经到达。一些数据源能提供严格的完整性界限，另一些只能依据可容忍乱序作估计；迟到事件可以违反这种估计。窗口分配、输出触发器、允许迟到范围与状态清理是不同决策，Flink 的时间模型展示了这些区分。
水位线经过窗口之前，提前给出结果也可能有用。当那笔 09:59 的购买到达时，系统可以在仍接受迟到数据的情况下修订旧总额，也可以送交单独核对，或按明确约定排除。更新结果需要消费者区分“替换后的总额”与“新增增量”，否则一次更正也可能被重复计数。
对于具有多个输入的算子，进展常受最慢的活跃输入限制。因此，即使其他分区很忙，空闲分区也可能延迟结果。空闲判定策略可以允许进展，但还需规定该分区恢复并送来旧事件时怎么办。应同时观察数据积压与水位线进展，不能把任务仍在运行等同于结果已经跟上数据。
Exactly-Once Semantics 与有状态处理
状态连接先后到达的事件：计数保留以前的递增，连接保留尚未匹配的记录，检测器保留模式中的前序步骤。故障恢复需要重新连接输入位置与状态。至多一次处理可能漏掉工作，至少一次处理可能重复工作。“恰好一次”语义描述规定范围内的效果，通常是对托管状态或事务输出的效果，而不是说代码在物理上只执行一次。
假设检查点记录计数 40 与输入位置 100。下一条记录把计数增加到 41，处理器随后在下一个检查点之前失败。一起恢复计数 40 与位置 100，再重放该记录，结果仍为 41。如果恢复计数 41，却又重放同一次递增，就会错误地得到 42。因此，状态与输入位置需要一致的恢复点。Flink 的状态模型介绍了检查点与重放的协调。
外部效果需要单独整合。即使内部状态正确，重放已经发送邮件或扣款的记录，也可能重复外部效果。事务型输出端，或与业务动作原子连接的持久去重，可以在其规定边界内建立更强保证。消息代理的事务不会自动包含任意外部 API。
检查点屏障标识处理图中的恢复边界，快照写入与对齐都有成本。数据源需要支持所需重放，状态与连接器配置也需要支持所声称的保证。如果恢复必须重现相同结果，还要处理非确定性计算与变化中的外部查询。
流连接除了键与时间，也必须规定保留策略。把订单与支付事件连接起来，可能需要在规定区间内保留尚未匹配记录。给旧订单补充客户地址，还需要选择使用当前地址，还是下单时有效的地址。这些属于查询语义，而不只是存储调优。
下游容量不足时，背压减慢上游生产或交付。缓冲可以吸收突发流量，却不能处理长期高于处理速率的到达速率。应跟踪队列增长、处理能力与中断后的恢复时间；要在持续接收新数据的同时追上积压，重放能力必须超过持续到达速率。
学习这一部分会改变什么
当流处理的响应时间或增量计算满足具体需求时，再选择它。如果允许数分钟延迟，周期性批查询可能更容易维护。无论采用哪种设计，都应规定结果的时间依据、新鲜度与更正处理方式。
可以用包含乱序与重复记录的小型销售流练习。先说明预期小时总额，再让处理器分别在检查点前后失败，分开检查内部状态与外部可见输出。这个练习比比较引擎功能列表，更直接地连接时间语义、重放与业务正确性。
资源
书籍与文本
Streaming Systems 解释时间、窗口、触发器与持续变化的结果。Kafka: The Definitive Guide 讲述特定日志平台，Designing Event-Driven Systems 把消息传递连接到应用架构，Stream Processing with Apache Flink 展开处理器专用机制。概念材料可以跨系统使用，但 API、连接器与恢复细节需要对照运行版本检查。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Akidau, Chernyak & Lax, Streaming Systems | 流处理中的时间与状态 | 入门 |
| Shapira、Palino、Sivaram 与 Petty，Kafka: The Definitive Guide（第 2 版） | Kafka 架构与生产运维 | 入门 |
| Stopford, Designing Event-Driven Systems（免费） | 事件驱动架构模式 | 深入 |
| Hueske & Kalavri, Stream Processing with Apache Flink | Flink 专门参考 | 参考 |
课程、论文与当前资料
Dataflow 模型论文 区分计算、窗口划分、触发与结果更新，Kreps 的 The Log 解释重放与独立消费。Flink、Kafka 与 Beam 文档规定可用机制，Confluent、Materialize 等工程文章提供实现案例。应区分文档保证、特定部署经验与产品比较。
| 资源 | 平台 | 类型 |
|---|---|---|
| Akidau et al., “The Dataflow Model”（VLDB 2015，免费） | 形式化基础 | 深入 |
| Kreps, “The Log” essay（免费） | 关于 event logs 的奠基文章 | 深入 |
| Apache Flink documentation（免费） | 权威有状态流处理参考 | 参考 |
| Apache Kafka documentation（免费） | 权威 Kafka 参考 | 参考 |
| Apache Beam documentation（免费） | 统一 programming model 参考 | 参考 |
| Confluent blog（免费） | 当代大规模流处理内容 | 参考 |
| Materialize blog（免费） | Streaming database 与 differential dataflow | 参考 |
实践、工具与当前资料
构造一个具有明确迟到策略的五分钟事件时间聚合。依次发送有序、乱序与状态清理后才到达的记录，验证预期输出，再中断处理并检查恢复。Kafka 消费者偏移与 Flink 界面帮助观察进展、检查点行为与背压。偏移积压衡量位置距离，并不直接衡量已经延迟多久，因此也要测量结果新鲜度。
| 资源 | 平台 | 类型 |
|---|---|---|
| Kafka Quickstart（免费） | kafka.apache.org | 实践 |
| Kafka Streams documentation（免费） | kafka.apache.org/documentation/streams | 参考 |
| Flink Quickstart（免费） | flink.apache.org | 实践 |
| Kafka consumer lag monitoring | kafka-consumer-groups.sh | 实践 |
| Flink web UI | local Flink cluster | 实践 |
| Apache Beam Playground（免费） | beam.apache.org | 实践 |
| Confluent Developer Kafka and Flink tutorials（免费；cloud features 可能付费） | developer.confluent.io | 实践 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 把 Kafka 等同于流处理 | 日志提供存储与消费机制，查询仍需要时间和状态语义。 | 先独立定义计算，再映射到消息代理与处理器能力。 |
| 不明确时间的含义 | 事件时间与处理时间可以把同一事件放入不同窗口。 | 按明确预期结果测试延迟和乱序事件。 |
| 认为恰好一次覆盖所有效果 | 状态恢复不会自动撤销或去重外部动作。 | 规定保证边界，在故障后同时检查内部状态与外部输出。 |
| 忽略模式演进 | 保留事件与并存的消费者版本可能采用不同模式。 | 规定兼容要求，测试代表性的新旧记录。 |
| 没有需要就采用事件溯源 | 重放与审计收益需要保留、版本管理及受控效果。 | 按所需历史与恢复行为评估这些成本。 |
| 把任务运行当作结果及时 | 活着的进程可能掩盖积压、停滞水位线与增长状态。 | 跟踪新鲜度、积压、检查点健康与追赶积压的可用容量。 |