English
Data engineering builds the processes that collect, transform, store, and deliver data for use. Data science formulates questions, examines evidence, builds models, and evaluates conclusions. They meet at the meaning of a record: what happened, to whom, when, and under which collection procedure?
Consider a dashboard reporting completed orders. The total depends on whether retries create duplicates, refunds revise old records, time zones change the reporting day, and “completed” has the same meaning across systems. Correct SQL cannot settle an undefined business concept. A useful data product needs both a reliable computation and an explicit interpretation.
The work connects databases (§4.4), distributed processing (§4.6–4.8), statistics (§2.5), and machine learning (§5.2). Data may be transformed before or after storage, and analysis may inform changes to the collection process itself. The goal is evidence suitable for a particular decision, with its provenance and limitations available for inspection.
Prerequisites: SQL and relational concepts (§4.4), basic algorithms (§2.6), and probability and statistics (§2.5). Begin with a small dataset whose records you can inspect before introducing distributed infrastructure.
From Statistical Analysis to Reproducible Data Pipelines
Data processing has roots in censuses, commercial record keeping, scientific measurement, and statistical work before electronic computers. Tabulation already required decisions about categories, units, missing records, and checks. As the computing histories show, business data processing and scientific calculation placed different demands on storage, input, and repeated operations. Bigger machines did not remove the need to define what was being counted.
Statistical methods developed ways to separate variation from evidence about populations and interventions. Interactive computing then made repeated fitting, plotting, and exploration easier. Tukey’s exploratory data analysis emphasized looking for unexpected structure and collection problems. Exploration can change the question; confirmatory analysis must account for which hypotheses and choices were selected after inspecting the data.
S and later R made statistical models, vectors, and graphics accessible in interactive environments. Python’s array, plotting, tabular-data, and notebook tools connected analysis with general programming. These ecosystems reduced the effort of expressing an analysis. They did not make package defaults appropriate for every sampling process, nor did they make an executable notebook automatically reproducible.
Relational databases and data warehouses organized durable records and analytical queries. At web scale, MapReduce provided a model in which map operations emit key–value pairs and reduce operations combine grouped values, with the framework managing distributed execution and recovery. Shuffling keys between workers can dominate cost. Partitioning and reducing data before a shuffle often matter more than increasing worker count.
Spark began at Berkeley in 2009 and was open-sourced in 2010, according to its project history. Reusing distributed datasets helped iterative and interactive workloads avoid repeatedly reconstructing intermediates. Its execution still includes scheduling, shuffles, serialization, and possible disk spill; “in memory” is a design opportunity, not a promise that a workload fits in RAM or is always faster.
Event logs and stream processors made arrival order and processing delay explicit design issues. Lambda architectures maintain batch and speed paths with a serving layer; Kappa-style designs emphasize processing and replaying a log through one main path. A log by itself is neither architecture. Each choice must explain corrections, late events, historical recomputation, and the cost of maintaining consistent definitions.
Cloud warehouses, transformation tools, and orchestrators changed where computation runs and how it is managed. Versioned SQL and dependency graphs make some changes easier to test and inspect. They complement rather than replace decisions about data ownership, access, retention, and semantics. No single tool combination is required for every analytical workload.
Machine-learning pipelines added requirements such as reproducing the features available at prediction time. Feature stores and model-tracking systems help organize that work, but consistency requires more than using the same column name. A training feature computed from information that arrived after the prediction would have been made can leak future information, even when the transformation code is shared.
Pipelines, Quality, and the Analytics Stack
Data Pipelines: From Source to Sink
Specify a pipeline’s sources, outputs, record identity, time semantics, and recovery behavior. Delivery can be at least once while an output is made effectively once through deduplication and atomic updates. An “exactly once” claim must state its boundary: a processor’s state, a database write, and an external notification need not share the same guarantee.
Suppose events e1:+10 and e2:+20 are followed by a retry of e1:+10. Blindly adding every arrival gives 40 instead of 30. Persisting the event ID and applying its effect in one transaction can prevent that duplicate effect. Recording the ID before a non-atomic effect can instead lose work after a crash. Idempotency is about repeated effects under a specified operation and input, not merely the presence of an upsert command.
ETL transforms before loading into the destination; ELT loads first and transforms there. Choose where to validate, minimize, and transform data based on privacy requirements, source constraints, compute, and downstream use. Raw retention can aid replay, but needs controlled access and a retention policy. A reproducible recomputation identifies input versions, transformation code, and parameters.
Separate event time from processing time. An event occurring at 23:59 may arrive at 00:03, so a daily total needs a rule for late data and revision. A watermark is an estimate or policy about event-time progress, not a universally reliable statement that every earlier event has arrived. For ML features, track when information became available as well as when it describes the world.
An orchestrator schedules dependencies and retries; it does not make task side effects correct. A backfill should recompute a specified historical interval without corrupting newer results. Version guards, partition replacement, transactional publication, and explicit failure handling can help, depending on the destination. Inspect the complete retry path, including what happens after the output commits but before success is acknowledged.
Data Quality: Measuring and Maintaining Trust
Define quality relative to use. Completeness, freshness, validity, uniqueness, consistency, and accuracy are different properties. A field can satisfy its type and range while describing the wrong person. A table can contain no nulls while systematically excluding failed transactions. Checks need an account of which records should exist, not only rules for those that arrived.
The grain of a table specifies what one row represents. Suppose two orders have totals 100 and 60; the first has two line items and the second has one. Joining order totals to line items and summing the repeated total gives 260, although the order-level total is 160. No row is necessarily malformed. The error comes from changing the unit of observation without changing the aggregation.
Use schema checks, key and relationship checks, reconciliation against independent counts or totals, and distribution monitoring. An unusual change may be a data defect or a real event. A fixed “within 20% of last week” check requires context such as holidays or a changed product. Route invalid records and alerts to an owner with a correction process rather than silently discarding inconvenient data.
Lineage connects outputs to input versions and transformations, helping locate where a meaning or value changed. Data contracts specify expectations between producers and consumers: fields, units, identity, timing, allowed corrections, and change notification. Automatically collected lineage and declared contracts are only as complete as the covered systems. Document the definition and owner of important metrics alongside their implementation.
Analytical Methods: From Exploration to Causation
Exploration examines distributions, missingness, unusual records, and relationships before committing to an interpretation. Preserve the distinction between observations that suggested a hypothesis and evidence reserved to test it. Removing an outlier requires a reason about the measurement or population; an inconvenient value is not automatically an error.
Randomized experiments assign an intervention at a defined unit, such as a user or household. The resulting comparison estimates an effect for that experiment’s population, exposure, and period. Randomization balances confounders in expectation, not exactly in every finite sample. Check assignment, actual exposure, missing outcomes, and interactions between units before treating a difference as a causal estimate.
Suppose 50 of 1,000 control users and 60 of 1,000 treatment users complete a task. The observed increase is 1 percentage point, or 20% relative to the control rate. With independent users, a simple normal-approximation 95% interval for the difference is roughly −1 to +3 percentage points. The relative increase alone exaggerates how much is known. Report the absolute effect, uncertainty, and what size would matter in practice.
A p-value measures incompatibility with a specified null model under the analysis assumptions; it is not the probability that the null is true or that chance caused the result. Plan sample size and stopping around the effect and precision needed. Repeatedly checking a fixed-sample significance threshold and stopping when it is crossed changes the error rate. Valid sequential designs can allow ongoing monitoring when their rules are followed.
Observational causal methods replace randomization with other assumptions. Difference-in-differences relies on a suitable parallel-trends claim; regression discontinuity uses behavior near an assignment threshold; instrumental variables require relevance and restrictions on alternative causal paths. These assumptions concern how the data arose, not just which regression was fitted. Domain knowledge, negative controls, and sensitivity analyses can assess parts of the argument without proving every untestable assumption.
Prediction, description, and intervention answer different questions. A model can predict cancellations accurately using a variable that would not reduce cancellations if changed. Keep population selection, measurement, data processing, and statistical inference visible in the final conclusion. More rows reduce some sampling uncertainty; they do not automatically repair selection bias or an invalid causal design.
What Studying This Changes
You should be able to state a metric’s population, row grain, time window, inclusion rules, and revision policy. Trace one reported value through its transformations to the source records. If its meaning cannot be stated, a faster pipeline will not resolve the problem.
Build a small order pipeline containing duplicate deliveries, late events, corrections, and one-to-many joins. Retry after a simulated failure and reconcile the result against a manually computed reference. Then construct a feature using only information available at a specified prediction time.
For an analysis, write the question and intended decision before choosing a method. Compare effect sizes and uncertainty, and separate exploratory findings from confirmatory evidence. This joins operational reliability with statistical reasoning: the computation and the interpretation both need support.
Resources
Books and Texts
Reis and Housley’s Fundamentals of Data Engineering maps the data lifecycle; Designing Data-Intensive Applications explains storage, distributed execution, and recovery. For analysis, use Tukey for exploration and Gelman/Hill for modeling and model checks. Cunningham’s Causal Inference: The Mixtape supplies worked causal designs; Imbens/Rubin develops their formal basis. Choose the book for the question, since operating a pipeline and identifying a causal effect require different arguments.
| Book | Role | Type |
|---|---|---|
| Kleppmann, Designing Data-Intensive Applications | Distributed foundations of data engineering | Depth |
| Reis & Housley, Fundamentals of Data Engineering | Comprehensive data engineering ecosystem | Entry |
| Tukey, Exploratory Data Analysis | Foundational data analysis philosophy | Entry |
| Gelman & Hill, Data Analysis Using Regression and Multilevel Models | Applied statistical modeling | Depth |
| Cunningham, Causal Inference: The Mixtape (free) | Causal inference methods for practitioners | Depth |
| Imbens & Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences | Formal causal inference | Depth |
| Huyen, Designing Machine Learning Systems | ML infrastructure and data pipelines | Depth |
Courses and Lectures
Data Engineering Zoomcamp, dbt Learn, and the orchestration tutorials offer implementation projects. Stanford CS246 goes deeper into large-data algorithms, while Missing Semester develops supporting command-line and tooling skills. Reproduce one complete local pipeline before adding a cloud account or cluster. Retain the schema, sample inputs, and expected results so that changing tools does not change the question being tested.
| Course | Platform | Type |
|---|---|---|
| Stanford CS 246 Mining Massive Datasets (free) | Stanford / YouTube | Depth |
| MIT Missing Semester (free) | missing.csail.mit.edu; supporting tooling | Auxiliary |
| dbt Learn (free) | courses.getdbt.com | Practice |
| Kaggle Learn (free) | kaggle.com/learn | Entry |
| DataTalksClub Data Engineering Zoomcamp (free) | DataTalksClub / GitHub / YouTube | Practice |
| Dagster University and documentation (free) | dagster.io / docs.dagster.io | Practice |
Practice, Tools, and Current Sources
Use DuckDB, pandas, or Polars to inspect transformations and join cardinalities on a small dataset. Arrow and Parquet documentation explain different aspects of columnar data interchange and storage. Add quality assertions and lineage before scaling the workload. For Spark or stream processing, use the project’s documentation to inspect partitions, shuffles, retries, and output guarantees; a managed platform does not remove those semantics.
| Resource | Platform | Type |
|---|---|---|
| DuckDB (free) | duckdb.org | Practice |
| dbt Core (free) | dbt-core / docs.getdbt.com | Practice |
| Great Expectations (free / paid cloud) | docs.greatexpectations.io | Practice |
| Apache Arrow columnar format (free) | arrow.apache.org | Reference |
| Apache Parquet documentation (free) | parquet.apache.org | Reference |
| Polars user guide (free) | docs.pola.rs | Practice |
| DataHub metadata platform (free / paid cloud) | docs.datahub.com | Reference |
| PySpark documentation and local examples (free) | spark.apache.org/docs | Practice |
| Apache Kafka documentation (free) | kafka.apache.org | Practice |
| Jupyter + pandas + matplotlib (free) | jupyter.org | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Assuming valid rows imply a valid metric | Joins, missing populations, time definitions, and aggregation can change meaning. | Specify grain and population, then reconcile the final result. |
| Claiming exactly once without a boundary | Processing, committing output, and external effects may have different failure behavior. | Test retries and crashes across the complete write path. |
| Using a large stack before measuring the workload | Scheduling and data movement can cost more than local computation. | Measure a simple baseline and identify the resource that requires scaling. |
| Treating statistical significance as importance or truth | It neither measures effect size nor gives the probability that a claim is true. | Report effect estimates, intervals, design assumptions, and practical relevance. |
| Inferring an intervention from a predictive association | Selection and common causes can produce the association. | Identify the causal design and justify its assumptions. |
| Using future-available data in historical training examples | A record may describe an earlier event but arrive after the prediction time. | Reconstruct both event time and information availability. |
中文
数据工程构建收集、变换、存储和交付数据的过程。数据科学提出问题、检查证据、建立模型并评价结论。两者在一条记录的含义上交汇：发生了什么、涉及谁、何时发生，又通过什么过程被记录？
考虑一个报告已完成订单的看板。总数取决于重试是否产生重复、退款是否修订旧记录、时区是否改变统计日期，以及各系统对“完成”的定义是否一致。正确的 SQL 不能解决尚未定义的业务概念。有用的数据产品既需要可靠计算，也需要明确解释。
这些工作连接数据库（§4.4）、分布式处理（§4.6–4.8）、统计（§2.5）和机器学习（§5.2）。数据可以在存储前或存储后变换，分析也可能反过来促使收集过程改变。目标是提供适合特定决策的证据，并让人能检查其来源和局限。
前置知识：SQL 与关系模型（§4.4）、基本算法（§2.6）、概率与统计（§2.5）。先从能够逐条检查的小数据集开始，再引入分布式基础设施。
从统计分析到可复现的数据流水线
数据处理的来源包括电子计算机出现前的人口普查、商业记录、科学测量与统计工作。制表已经需要决定类别、单位、缺失记录的处理和检查方式。计算史展示了商业数据处理与科学计算对存储、输入和重复操作的不同要求。机器更大，也不能免去定义统计对象的工作。
统计方法研究怎样从变异中获得关于总体和干预的证据。交互计算随后使反复拟合、绘图和探索更容易。Tukey 的探索性数据分析强调发现意外结构与收集问题。探索可以改变问题，而验证性分析需要考虑哪些假设和选择是在看过数据之后才确定的。
S 和后来的 R 将统计模型、向量和图形带入交互环境。Python 的数组、绘图、表格数据和笔记本工具，将分析与通用编程联系起来。这些生态降低了表达分析过程的成本，但不会使包的默认设置适合每种采样过程，也不会让可执行笔记本自动具备可复现性。
关系数据库和数据仓库组织持久记录与分析查询。在 Web 规模下，MapReduce 提供了这样的模型：map 操作输出键值对，reduce 操作合并按键分组的值，框架负责分布式执行和恢复。工作节点之间的键重分布可能成为主要成本。在重分布前做好分区、减少数据，往往比增加节点数量更重要。
根据 Spark 的项目历史，它于 2009 年始于 Berkeley，2010 年开源。复用分布式数据集，帮助迭代和交互工作负载避免反复重建中间结果。执行中仍有调度、重分布、序列化，也可能溢写磁盘；“内存计算”是一种设计机会，不是数据一定装得进内存或总会更快的保证。
事件日志和流处理器，使到达顺序与处理延迟成为明确的设计问题。Lambda 架构维护批处理路径、快速路径和服务层；Kappa 式设计强调通过一条主要路径处理及重放日志。仅有日志不等于其中任何一种架构。每种选择都需要解释如何处理修订、迟到事件、历史重算，以及保持定义一致的成本。
云数据仓库、变换工具和编排器，改变了计算的运行位置与管理方式。版本化 SQL 和依赖图，使部分变更更容易测试和检查。它们不能替代对数据所有权、访问、保留和语义的决定，也不存在每种分析工作负载都必须使用的工具组合。
机器学习流水线增加了复现预测时可用特征等要求。特征库和模型跟踪系统有助于组织这些工作，但一致性不只是在使用相同列名。如果某个训练特征使用了本应预测之后才到达的信息，即使共用了变换代码，仍可能泄漏未来信息。
流水线、质量与分析栈
数据流水线：从来源到去向
应明确流水线的来源、输出、记录标识、时间语义与恢复行为。消息可以至少交付一次，同时通过去重和原子更新，使输出效果只发生一次。“恰好一次”的主张必须说明边界：处理器内部状态、数据库写入和外部通知，未必共享同一保证。
假设事件依次为 e1:+10、e2:+20，随后重试 e1:+10。每次到达都直接累加，会得到 40，而不是 30。在同一事务中持久保存事件 ID 并应用其效果，可以避免重复效果。如果先记下 ID，再非原子地执行效果，崩溃后反而可能漏做。幂等性描述指定操作和输入下的重复效果，不是代码中出现了 upsert 就自动成立。
ETL 在载入目标系统前变换，ELT 则先载入，再在目标系统中变换。根据隐私要求、来源约束、算力与后续用途，决定在哪里验证、最小化和变换数据。保留原始数据有助于重放，但需要访问控制和保留策略。可复现的重算应确定输入版本、变换代码和参数。
区分事件时间与处理时间。23:59 发生的事件可能在 00:03 才到达，因此每日汇总需要迟到数据与修订规则。水位线是关于事件时间进度的估计或策略，不是所有更早事件必然已经到达的可靠声明。机器学习特征不仅要记录信息描述的时间，也要记录信息何时实际可用。
编排器调度依赖和重试，但不会使任务副作用自动正确。历史回填应重算指定区间，同时避免破坏更新的结果。版本检查、分区替换、事务性发布和明确的失败处理，可以按目标系统的能力发挥作用。需要检查整个重试路径，包括输出已提交、成功确认尚未返回时会发生什么。
数据质量：衡量和维护信任
质量需要相对于用途定义。完整性、新鲜度、有效性、唯一性、一致性和准确性是不同性质。字段可能类型、范围都合规，却描述了错误的人；表中可能没有空值，却系统性排除了失败交易。检查不仅需要约束已到达的记录，也需要说明哪些记录本来应该存在。
表的粒度说明一行代表什么。假设两个订单总额分别为 100 和 60，第一个有两条明细，第二个有一条。把订单总额连接到明细后，再累加被重复的总额，会得到 260，而订单层面的总额是 160。单条记录未必有格式错误；问题来自观察单位改变后，聚合方式没有相应改变。
结合模式检查、键与关系检查、独立数量或总额对账，以及分布监控。异常变化可能是数据缺陷，也可能是真实事件。“与上周相比变化不超过 20%”这样的固定检查，需要考虑节假日或产品变化等背景。异常记录与告警应有负责人和纠正流程，不应悄悄丢弃不便处理的数据。
血缘将输出与输入版本和变换关联起来，帮助定位含义或值在哪里改变。数据契约明确生产者与消费者之间的约定：字段、单位、标识、时间、允许的修订和变更通知。自动收集的血缘与声明的契约，只能覆盖实际纳入的系统。重要指标应同时记录定义、负责人和实现。
分析方法：从探索到因果
探索分析检查分布、缺失、异常记录和变量关系，再形成解释。应区分提出假设的观察与保留用于检验假设的证据。删除离群值，需要关于测量或总体的理由；结果不合预期，不代表记录必然错误。
随机实验在明确单位上分配干预，例如用户或家庭。比较估计的是该实验人群、接触条件和时间段中的效应。随机化在期望上平衡混杂因素，而不是在每个有限样本中都精确平衡。在把差异视为因果估计前，先核对分配、实际接触、结果缺失和单位之间的影响。
假设 1,000 名对照用户中有 50 人完成任务，1,000 名处理用户中有 60 人完成。观察到的提升是 1 个百分点，相对对照完成率提升 20%。若用户独立，用简单正态近似计算，差值的 95% 区间约为 −1 到 +3 个百分点。只报告相对提升，会夸大已知信息。应同时报告绝对效应、不确定性，以及多大的变化具有实际意义。
p 值衡量的是在分析假设下，数据与指定零假设模型的不相容程度；它不是零假设为真的概率，也不是结果由偶然造成的概率。根据所需效应和精度规划样本规模与停止规则。反复查看固定样本检验的显著性阈值，一旦越过就停止，会改变错误率。遵循适当规则的序贯设计，则可以支持持续监测。
观察性因果方法用其他假设替代随机化。双重差分依赖适当的平行趋势主张，回归不连续利用分配阈值附近的行为，工具变量需要相关性及对其他因果路径的限制。这些假设涉及数据怎样产生，而不只是拟合了哪一种回归。领域知识、负对照和敏感性分析可以检查部分论证，但不能证明所有不可检验假设。
预测、描述和干预回答不同问题。模型可能利用某个变量准确预测取消行为，但改变该变量未必减少取消。最终结论应让人看清人群选择、测量、数据处理与统计推断。更多记录可以减少部分采样不确定性，但不会自动修复选择偏差或无效的因果设计。
学习这一部分会改变什么
学完后，应能说明一个指标的人群、行粒度、时间窗口、纳入规则和修订策略。沿变换过程把一个报告值追溯到来源记录。如果含义说不清，更快的流水线也解决不了问题。
构建一个包含重复交付、迟到事件、修订和一对多连接的小型订单流水线。模拟失败后重试，并与手算参考结果对账。再只使用指定预测时间之前已可获得的信息构造一个特征。
分析时，在选择方法之前写出问题和预期决策。比较效应大小与不确定性，区分探索发现和验证证据。这样把运行可靠性与统计推理联系起来：计算过程和结果解释都需要依据。
资源
书籍与文本
Reis 与 Housley 的 Fundamentals of Data Engineering 介绍数据生命周期，Designing Data-Intensive Applications 解释存储、分布式执行和恢复。分析方面，可以用 Tukey 学习探索，用 Gelman/Hill 学习建模与模型检查。Cunningham 的 Causal Inference: The Mixtape 提供具体因果设计，Imbens/Rubin 展开其形式基础。应按问题选书，因为运行流水线与识别因果效应需要不同论证。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Kleppmann, Designing Data-Intensive Applications | 数据工程的分布式基础 | 深入 |
| Reis & Housley, Fundamentals of Data Engineering | 综合数据工程生态 | 入门 |
| Tukey, Exploratory Data Analysis | 基础数据分析哲学 | 入门 |
| Gelman & Hill, Data Analysis Using Regression and Multilevel Models | 应用统计建模 | 深入 |
| Cunningham, Causal Inference: The Mixtape（免费） | 面向实践者的因果推断方法 | 深入 |
| Imbens & Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences | 正式因果推断 | 深入 |
| Huyen, Designing Machine Learning Systems | ML 基础设施与数据流水线 | 深入 |
课程与讲座
Data Engineering Zoomcamp、dbt Learn 和编排教程提供实现项目。Stanford CS246 更深入地研究大数据算法，Missing Semester 补充命令行与工具技能。先复现一个完整本地流水线，再按需要使用云账户或集群。保存模式、样例输入和预期结果，使更换工具不会改变正在检验的问题。
| 课程 | 平台 | 类型 |
|---|---|---|
| Stanford CS 246 Mining Massive Datasets（免费） | Stanford / YouTube | 深入 |
| MIT Missing Semester（免费） | missing.csail.mit.edu；配套工具技能 | 辅助 |
| dbt Learn（免费） | courses.getdbt.com | 实践 |
| Kaggle Learn（免费） | kaggle.com/learn | 入门 |
| DataTalksClub Data Engineering Zoomcamp（免费） | DataTalksClub / GitHub / YouTube | 实践 |
| Dagster University and documentation（免费） | dagster.io / docs.dagster.io | 实践 |
实践、工具与当前资料
用 DuckDB、pandas 或 Polars，在小数据集上检查变换和连接后的行数。Arrow 与 Parquet 文档分别解释列式数据交换和存储的相关问题。在扩展工作负载之前，先加入质量断言和血缘记录。学习 Spark 或流处理时，使用项目文档检查分区、重分布、重试与输出保证；托管平台不会消除这些语义。
| 资源 | 平台 | 类型 |
|---|---|---|
| DuckDB（免费） | duckdb.org | 实践 |
| dbt Core（免费） | dbt-core / docs.getdbt.com | 实践 |
| Great Expectations（免费 / 付费 cloud） | docs.greatexpectations.io | 实践 |
| Apache Arrow columnar format（免费） | arrow.apache.org | 参考 |
| Apache Parquet documentation（免费） | parquet.apache.org | 参考 |
| Polars user guide（免费） | docs.pola.rs | 实践 |
| DataHub metadata platform（免费 / 付费 cloud） | docs.datahub.com | 参考 |
| PySpark 文档与本地例子（免费） | spark.apache.org/docs | 实践 |
| Apache Kafka 文档（免费） | kafka.apache.org | 实践 |
| Jupyter + pandas + matplotlib（免费） | jupyter.org | 实践 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 认为记录合规就代表指标正确 | 连接、缺失人群、时间定义和聚合都可能改变含义。 | 定义粒度与人群，再核对最终结果。 |
| 不说明边界就宣称恰好一次 | 处理、提交输出和外部效果可能有不同失败行为。 | 测试整个写入路径上的重试与崩溃。 |
| 未测工作负载就使用庞大工具栈 | 调度和数据移动可能比本地计算更昂贵。 | 测量简单基线，找出确实需要扩展的资源。 |
| 把统计显著当作重要性或真理 | 它不衡量效应大小，也不给出主张为真的概率。 | 报告效应估计、区间、设计假设与实际意义。 |
| 从预测关联直接推出干预效果 | 选择和共同原因都可能产生关联。 | 明确因果设计，并论证其假设。 |
| 在历史训练样本中使用未来才可用的数据 | 记录可能描述早期事件，却在预测时间之后才到达。 | 同时重建事件时间与信息可用时间。 |