English
Data is not facts; it is recorded observations, with all the messiness that implies. Real data arrives late, arrives duplicated, arrives with wrong values, arrives in formats that do not match what was agreed, arrives from systems that crashed mid-write, and sometimes does not arrive at all. Data engineering is the discipline of building the infrastructure that moves data from where it originates to where it can be used — reliably, at scale, with documented lineage, and with the quality guarantees that downstream consumers need. Data science is the discipline of extracting understanding from data: asking questions that data can answer, designing analyses that answer those questions without being fooled by noise or bias, and communicating conclusions in ways that distinguish what the data supports from what it does not.
The two disciplines are deeply coupled. A data scientist who cannot trust the pipeline feeding their analysis is building on sand; a data engineer who does not understand what the data will be used for builds infrastructure that is technically correct but analytically useless. The clearest way to express the boundary: data engineering asks “how do we get clean, reliable data to the right place at the right time?”; data science asks “what does the data tell us, and how certain can we be?” Both questions must be answered before data generates value.
This section sits between databases (§4.4) and machine learning (§5.2). Databases provide the storage and query infrastructure; this section addresses what happens before data reaches a database (ingestion, transformation, quality), what analytics can be done with it (exploratory analysis, A/B testing, causal inference), and how it is prepared for modeling. Machine learning provides the predictive modeling methods; this section addresses the feature engineering, experiment tracking, and data infrastructure that make those methods practically applicable at scale.
Prerequisites: Databases (§4.4) — SQL and basic relational concepts are used throughout. Probability and statistics (§2.5) — distributions, confidence intervals, hypothesis testing, and causation vs. correlation are the analytical foundation. Algorithms (§2.6) — understanding cost models and data structure tradeoffs is necessary for reasoning about pipeline performance.
How Statistics Met Computation and What the Collision Produced
The history of data science and data engineering is a collision between two traditions that developed independently for most of the twentieth century: statistical theory and computational infrastructure.
Statistical theory grew from the work of Fisher, Neyman, Pearson, and their successors in the first half of the twentieth century, primarily on small datasets collected by deliberate experiment or careful survey. The practice was manual computation or at best mechanical calculators; a dataset with hundreds of observations was large. The intellectual framework was precise and demanding: hypothesis testing, confidence intervals, experimental design, analysis of variance. The assumptions were explicit and the methods were validated against those assumptions. The scale was human-manageable.
Computational statistics arrived when computers made larger datasets tractable. John Tukey’s 1977 book Exploratory Data Analysis introduced a practice and a philosophy: before testing hypotheses, look at the data. Plot the distributions. Find the outliers. Look for structure you did not anticipate. Tukey argued that the hypothesis-testing framework, while valuable, was applied prematurely in much scientific practice — that the appropriate first step was exploration, which might reveal that the hypothesis was misspecified, the data collection was flawed, or the interesting question was different from the one initially posed. EDA remains the correct starting point for any data analysis, and its centrality to data science practice traces to Tukey.
The language that brought statistical computation to practitioners was S, developed at Bell Labs by John Chambers and colleagues beginning in 1976. S provided an interactive environment for data analysis, with vectors and matrices as first-class objects, formula notation for statistical models, and a graphics system for visualization. S was commercialized as S-Plus; its open-source reimplementation as R, begun by Ross Ihaka and Robert Gentleman at the University of Auckland in the early 1990s, became the dominant platform for statistical computing in academia and much of industry. R’s package ecosystem (CRAN), which had accumulated over 15,000 packages by 2020 covering nearly every statistical method, represents one of the most ambitious collaborative software projects in science.
Python entered the data analysis space more gradually, through NumPy (2006), which provided efficient array operations; pandas (2008), which provided labeled data frames modeled loosely on R’s data frame concept; and matplotlib (2003) for visualization. The combination — coupled with scikit-learn for machine learning and Jupyter notebooks for interactive computation — made Python the dominant language for data science by the early 2010s, particularly in industry where the ability to integrate with production software systems mattered more than the depth of the statistical package ecosystem.
The computational scale problem arrived with the web. By the early 2000s, companies like Google and Yahoo were accumulating data volumes — logs, user behavior, crawl results — that could not be processed on a single machine. Google’s 2004 paper “MapReduce: Simplified Data Processing on Large Clusters” described a programming model for distributed data processing: a user-defined Map function transforms each record, a shuffle phase groups transformed records by key, and a user-defined Reduce function aggregates each group. The framework handles the distribution, fault tolerance, and parallelism automatically. MapReduce was elegant and powerful, and the Apache Hadoop project implemented it as open-source software in 2006. Hadoop, combined with HDFS (Hadoop Distributed File System), became the dominant platform for large-scale data processing and created the “big data” ecosystem of tools, companies, and practices.
Hadoop had fundamental limitations. MapReduce required writing intermediate results to disk between stages, making iterative algorithms (essential for machine learning and graph processing) extremely slow. SQL queries over Hadoop (through Hive) were orders of magnitude slower than SQL over relational databases. Programming the MapReduce API directly was verbose and error-prone. Apache Spark, developed at UC Berkeley’s AMPLab and open-sourced in 2012 (publicly released 2014), addressed these limitations through in-memory computation and a higher-level API. Spark’s RDD (Resilient Distributed Dataset) and later DataFrame and Dataset APIs made distributed data processing accessible to practitioners who could express transformations in Python, Scala, or Java without writing MapReduce directly. Spark became the dominant batch processing engine for large-scale data engineering and remains central to the ecosystem.
Streaming data processing emerged as a separate problem: not all analytics can wait for a batch job to complete. Monitoring dashboards, fraud detection, real-time recommendations, and many other applications require processing events within seconds or milliseconds of their occurrence. Apache Kafka, developed at LinkedIn and open-sourced in 2011, provided a distributed log that could store and serve event streams at high throughput. Apache Flink (2014) and Apache Storm (earlier) provided the stream processing engines. The architectural pattern — a durable log (Kafka) feeding both stream processors (for real-time analytics) and batch processors (for historical analytics) — became known as the Lambda architecture, though its complexity (maintaining two separate code paths) motivated the later Kappa architecture (treating batch as a special case of streaming).
The modern data stack emerged from dissatisfaction with the complexity of Hadoop-era infrastructure. Cloud data warehouses — Snowflake (2012), Google BigQuery (2010), Amazon Redshift (2012) — demonstrated that SQL-based analytics at scale could be much simpler than Hadoop if the compute was elastically scalable and the storage was cheap. dbt (data build tool, 2016) applied software engineering practices to data transformation: SQL transformations as version-controlled code, automated testing, documentation, and lineage tracking. Airflow (2014), Prefect, and Dagster provided workflow orchestration for data pipelines. The combination — cloud data warehouse, transformation tool, and orchestration — became the “modern data stack” that replaced most Hadoop deployments in industry by the early 2020s for analytics workloads.
The most recent development is the convergence of data engineering and machine learning. Feature stores (Feast, Hopsworks, Tecton) solve the data engineering problem specific to ML: features needed for model training and inference must be computed consistently, stored efficiently, served at low latency, and tracked for reproducibility. ML platforms (MLflow, Kubeflow, Vertex AI) add experiment tracking, model versioning, and deployment infrastructure. The “MLOps” movement applies DevOps practices to machine learning: continuous training pipelines, automated evaluation, deployment automation, monitoring for data drift and model degradation. These developments have made data engineering and ML infrastructure increasingly integrated, requiring practitioners who understand both.
Pipelines, Quality, and the Analytics Stack
Data Pipelines: From Source to Sink
A data pipeline is a sequence of transformations that moves data from sources — production databases, event streams, third-party APIs, file uploads — to sinks — data warehouses, feature stores, downstream applications. The pipeline’s responsibility is not just moving data but maintaining its correctness: ensuring that each record is processed exactly once, that failures are handled gracefully and operations are idempotent (can be retried without producing duplicate results), that schema changes in sources are handled without breaking downstream consumers, and that data arrives in a form that consumers can use.
The ELT pattern (Extract, Load, Transform) has displaced ETL (Extract, Transform, Load) in most modern analytics architectures. In ETL, data is transformed before loading into the data warehouse, which requires maintaining complex transformation logic outside the warehouse. In ELT, raw data is loaded into the warehouse first and transformed there using SQL. The shift reflects the economics of cloud storage (cheap enough to store raw data) and compute (elastic enough to transform large volumes in SQL without a separate processing infrastructure). dbt, the dominant transformation tool, implements ELT: it takes SQL SELECT statements as input and materializes them as tables or views in the warehouse, with dependency tracking, testing, and documentation.
Pipeline reliability requires thinking through failure modes: what happens if a source is unavailable? what happens if a record fails validation? what happens if a downstream consumer is slow? The idempotency requirement — that re-running a pipeline produces the same result as running it once — is the most important single design constraint. Pipelines that are not idempotent produce duplicate data when retried, and retrying is necessary when anything fails. Making operations idempotent often requires using upserts (insert-or-update) rather than inserts, tracking watermarks to know which records have been processed, and using transactional load patterns that are atomic with respect to the target system.
Data orchestration tools — Apache Airflow, Prefect, Dagster — schedule and coordinate pipeline execution. A DAG (directed acyclic graph) of tasks specifies dependencies: task B runs after task A, task C runs after both B and D. The orchestrator handles retries on failure, monitors task completion, sends alerts when pipelines are late, and provides visibility into historical runs. The choice between orchestrators involves tradeoffs between operational complexity (Airflow is powerful but operationally demanding), developer experience (Prefect and Dagster emphasize ease of use), and integration with the broader data stack.
Data Quality: Measuring and Maintaining Trust
Data quality is not a binary property. Data is not clean or dirty; it has measurable quality along multiple dimensions — completeness (are all expected records present?), freshness (how recent is the data?), accuracy (do values match their source of truth?), consistency (are related values consistent across tables?), uniqueness (are there unexpected duplicates?), and validity (do values conform to expected formats and ranges?). Each dimension can be measured, trended, and alerted on.
Data quality testing, popularized by dbt and tools like Great Expectations, applies software testing practices to data. Tests specify expectations about data: this column should never be null, this foreign key should always match its parent, this metric should be within 20% of last week’s value. Tests run automatically as data is loaded or transformed, catching quality issues before they propagate to downstream analyses. The shift from ad-hoc quality checks to systematic automated testing has been one of the most significant improvements in data engineering practice in the past decade.
Data lineage — tracking which tables a downstream table was derived from, and which transformations were applied — is essential for debugging quality issues. When a downstream metric changes unexpectedly, lineage allows engineers to trace the change to its source: which upstream table changed? which transformation applied to it? which source system is the root cause? Modern transformation tools (dbt) and metadata platforms (DataHub, OpenMetadata, Atlan) maintain lineage automatically for SQL-defined transformations, though lineage for custom code remains harder to capture.
Data contracts, a more recent practice, specify the interface between data producers and consumers: what fields the producer will supply, what their types will be, what quality guarantees hold, and what notice will be given before changes. Contracts formalize the implicit agreements that previously existed only in documentation or in engineers’ heads, making them testable and automatable. The data contract pattern addresses the fundamental friction in data engineering: data producers (product teams, operational systems) optimize for their own needs, and data consumers (analysts, data scientists, downstream applications) are broken when producer behavior changes without notice.
Analytical Methods: From Exploration to Causation
Exploratory data analysis is the first step in any serious data analysis. Before testing hypotheses or fitting models, the analyst examines the data: what is the distribution of each variable? what are the outliers and how should they be handled? what are the missing values and what do they imply? what correlations exist between variables? The exploration often reveals problems — data collection errors, unexpected seasonality, population heterogeneity — that invalidate the originally intended analysis and redirect the work toward the actual questions the data can answer.
A/B testing (randomized controlled experiments) is the gold standard for measuring causal effects in digital systems. Users are randomly assigned to control and treatment conditions; a metric of interest is measured for both groups; a statistical test determines whether the observed difference is likely due to the treatment or to chance. The randomization ensures that the only systematic difference between groups is the treatment, making causal inference valid. However, A/B tests have requirements: the treatment must be assignable at the unit of randomization, the treatment and control conditions must be well-defined, the metric must be measurable without contamination between groups, and sufficient sample size is required to detect the expected effect. Many business questions cannot be answered by A/B tests (historical comparisons, network effects, ethical constraints on randomization), requiring alternative approaches.
Causal inference without randomization — using observational data to estimate causal effects — is the hardest methodological problem in data science. The fundamental challenge is confounding: factors that affect both the treatment and the outcome, creating the appearance of a causal relationship where none exists. Difference-in-differences, instrumental variables, regression discontinuity, and synthetic control are the standard methods for handling specific confounding structures when randomization is not available. Each method requires assumptions that must be justified by domain knowledge and tested for plausibility; no method works universally. The practitioner who treats correlation as causation from observational data, without accounting for confounding, will systematically produce wrong conclusions.
Experiment design is not a post-hoc concern. The sample size needed to detect an effect of a given magnitude, at given significance and power levels, must be calculated before collecting data. A study with insufficient power will fail to detect real effects; a study that looks at the data and decides to continue collecting when results are not yet significant inflates the Type I error rate. Pre-registration — specifying the hypothesis, design, analysis plan, and stopping rules before seeing the data — is the practice that separates confirmatory from exploratory analysis. The replication crisis in social science, psychology, and medicine has demonstrated at scale what statisticians had always known: underpowered studies combined with flexible analysis decisions produce findings that do not replicate.
What Studying This Changes
Data engineering and data science change how practitioners think about the evidence behind claims and about the infrastructure that generates that evidence.
The first change is pipeline thinking: the instinct to trace any data product back to its sources and question whether the pipeline is reliable. A metric that looked good but was computed from data that arrived late, was improperly deduplicated, or was transformed with a bug is meaningless. The practitioner who has studied data engineering looks at any dashboard or analysis and immediately asks: where does this data come from? how is it transformed? how would I know if it were wrong? This instinct prevents the common failure mode of making decisions based on data that appears authoritative but is actually corrupted.
The second change is statistical discipline: the habit of distinguishing what data shows from what it proves. Correlation does not imply causation. Statistical significance does not imply practical significance. A result that is significant with n = 10,000 observations might be meaningless — the effect is real but tiny. A result that is not significant with n = 100 observations tells you almost nothing. The practitioner who has studied data science makes these distinctions automatically and is appropriately skeptical of claims that do not.
The third change is scale intuition: the ability to predict what a data processing approach will cost before building it. Processing 100 rows and processing 100 million rows require different approaches. A join that is trivial in SQL becomes a shuffle operation that moves terabytes across a network in a distributed system. An aggregation that takes milliseconds on a local machine takes hours if the data is unpartitioned and spread across many files. This cost intuition guides architectural choices and prevents the common failure of building systems that work in development and fail in production.
The fourth change is the ability to design experiments and interpret their results correctly. Not every claim about product performance, user behavior, or business metrics is backed by a well-designed experiment. The practitioner who understands experimental design can identify when a claimed causal relationship might be confounded, when a sample size is insufficient to detect a meaningful effect, and when the analysis choices made after seeing the data invalidated the statistical test. This ability is increasingly valuable as organizations make more decisions based on data.
Resources
Books and Texts
Kleppmann’s Designing Data-Intensive Applications (O’Reilly, 2017, referenced in §4.4) covers the distributed systems and database foundations of data engineering at depth. For data engineers specifically, the chapters on batch processing, stream processing, and distributed data systems are particularly relevant. It is the best single-volume treatment of the infrastructure that data pipelines run on.
Reis and Housley’s Fundamentals of Data Engineering (O’Reilly, 2022) is the most comprehensive treatment of data engineering as a discipline: the data engineering lifecycle (generation, ingestion, transformation, serving), the tools and platforms at each stage, and the decision-making frameworks for choosing between them. It is more prescriptive and less conceptually deep than Kleppmann but more comprehensive in covering the full ecosystem.
For data analysis methodology, Tukey’s Exploratory Data Analysis (Addison-Wesley, 1977) remains the foundational text. It is dated in its computational tools but not in its philosophy: look at the data before testing hypotheses, be suspicious of single-number summaries, seek to understand the structure before fitting a model. Any serious data practitioner should read it.
Gelman and Hill’s Data Analysis Using Regression and Multilevel/Hierarchical Models (Cambridge, 2007) is the best treatment of applied statistical modeling for data scientists. It covers regression, generalization of regression to hierarchical data structures, and model checking and validation, with a Bayesian perspective that makes uncertainty quantification natural. The practical examples throughout are from real social science research, making the methods concrete.
Cunningham’s Causal Inference: The Mixtape (Yale University Press, 2021, free online at mixtape.scunning.com) covers the major methods for causal inference from observational data — potential outcomes framework, difference-in-differences, instrumental variables, regression discontinuity, synthetic control — with worked examples in Stata and R. It is the most accessible entry to the causal inference literature for practitioners with a basic statistics background.
Imbens and Rubin’s Causal Inference for Statistics, Social, and Biomedical Sciences (Cambridge, 2015) provides the formal statistical treatment of causal inference. It is more demanding than Cunningham but more rigorous, and it is the right text for practitioners who want to understand why the methods work rather than just how to apply them.
For the ML data infrastructure side, Huyen’s Designing Machine Learning Systems (O’Reilly, 2022) covers feature engineering, training data, data and feature pipelines, and ML deployment with a systems perspective. It is the most useful single book for practitioners building ML infrastructure.
| Book | Role | Type |
|---|---|---|
| Kleppmann, Designing Data-Intensive Applications | Distributed foundations of data engineering | Depth |
| Reis & Housley, Fundamentals of Data Engineering | Comprehensive data engineering ecosystem | Entry |
| Tukey, Exploratory Data Analysis | Foundational data analysis philosophy | Entry |
| Gelman & Hill, Data Analysis Using Regression and Multilevel Models | Applied statistical modeling | Depth |
| Cunningham, Causal Inference: The Mixtape (free) | Causal inference methods for practitioners | Depth |
| Imbens & Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences | Formal causal inference | Depth |
| Huyen, Designing Machine Learning Systems | ML infrastructure and data pipelines | Depth |
Courses and Lectures
Stanford CS 246 (Mining Massive Datasets, materials and video free online) covers the algorithmic foundations of large-scale data processing: locality-sensitive hashing, dimensionality reduction, PageRank, recommendation systems, and distributed data mining algorithms. It bridges the gap between algorithm theory and the practical challenges of processing data at scale.
The Missing Semester of Your CS Education (MIT, free, missing.csail.mit.edu) covers the practical computing skills — shell scripting, version control, data wrangling with command-line tools — that form the daily toolkit of data engineers but are rarely taught in CS curricula. The data wrangling lecture (using sed, awk, grep, and related tools) is particularly valuable.
Kaggle Learn (free) offers structured notebooks on pandas, SQL, feature engineering, and data visualization. The tutorials are practical and interactive, appropriate as an introduction to the tools before deeper conceptual study.
dbt Learn (free, courses.getdbt.com) provides structured training on dbt fundamentals and advanced features. Given dbt’s centrality in the modern data stack, this is worth completing for any practitioner working with data warehouses.
| Course | Platform | Type |
|---|---|---|
| Stanford CS 246 Mining Massive Datasets (free) | Stanford / YouTube | Depth |
| MIT Missing Semester (free) | missing.csail.mit.edu | Entry |
| dbt Learn (free) | courses.getdbt.com | Practice |
| Kaggle Learn (free) | kaggle.com/learn | Entry |
| DataTalksClub Data Engineering Zoomcamp (free) | DataTalksClub / GitHub / YouTube | Practice |
| Dagster University and documentation (free) | dagster.io / docs.dagster.io | Practice |
Practice, Tools, and Current Sources
The data engineering toolkit is primarily tools rather than visualizations. The essential tools to learn are:
SQL on a data warehouse is the most important skill. Write SQL in Snowflake, BigQuery, or DuckDB against real datasets; use EXPLAIN to see query plans; observe how query performance changes with different join orders, with and without clustering or partitioning. DuckDB (free, runs locally) is the best learning environment: it processes local CSV and Parquet files at high speed with a full SQL implementation, including window functions, lateral joins, and other advanced features.
dbt with a local or cloud data warehouse provides the transformation layer. The dbt tutorial (free at docs.getdbt.com/docs/get-started/dbt-core-quickstart) walks through creating models, tests, and documentation from a sample dataset. Extending the tutorial with custom models and tests is the right follow-on.
Apache Spark via PySpark for large-scale batch processing. The Databricks Community Edition (free) provides a managed Spark environment with notebooks. Processing a dataset that is too large for local pandas — several hundred million rows — and observing the performance differences between different partition strategies, join algorithms, and aggregation approaches builds scale intuition.
Apache Kafka for event streaming. Confluent’s free tier and tutorial provide a managed Kafka environment. The tutorial for a streaming data pipeline — producing events, consuming them, applying transformations — makes the streaming paradigm concrete.
Jupyter notebooks with pandas, matplotlib, and seaborn for exploratory data analysis. The most educational exercise is taking a real dataset, performing EDA without a predetermined hypothesis, documenting what is found, and then designing an analysis based on the exploration. The UCI Machine Learning Repository and Kaggle datasets both provide suitable starting points.
| Resource | Platform | Type |
|---|---|---|
| DuckDB (free) | duckdb.org | Practice |
| dbt Core (free) | dbt-core / docs.getdbt.com | Practice |
| Great Expectations (free / paid cloud) | docs.greatexpectations.io | Practice |
| Apache Arrow columnar format (free) | arrow.apache.org | Reference |
| Apache Parquet documentation (free) | parquet.apache.org | Reference |
| Polars user guide (free) | docs.pola.rs | Practice |
| DataHub metadata platform (free / paid cloud) | docs.datahub.com | Reference |
| PySpark on Databricks Community Edition (free) | databricks.com | Practice |
| Apache Kafka / Confluent free tier | confluent.io | Practice |
| Jupyter + pandas + matplotlib (free) | jupyter.org | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Trusting the pipeline without verifying it | A metric that looks reasonable but is computed from corrupted data produces incorrect conclusions with high confidence. Data quality problems are pervasive and often invisible: late data that appears as if records were missing, deduplication bugs that double-count events, timezone mishandling that shifts dates by one day, schema changes that silently break downstream transformations. Practitioners who do not verify data quality build analyses on unstable ground. | Before relying on any dataset for important decisions, trace the pipeline that produced it: what is the source? what transformations were applied? where could duplicates enter? are there automated quality tests? Run basic sanity checks on every dataset before analysis: row counts by date, null rates for key columns, range checks on numeric values. Treat data quality as the first step of every analysis. |
| Conflating correlation with causation | Observational data shows correlations; establishing causation requires additional assumptions or experimental design. Users who spent more time on a website before conversion did not necessarily convert because they spent more time — users who were going to convert anyway might browse more. Confounding is ubiquitous in behavioral data, and analyses that treat observational correlations as causal relationships systematically produce wrong conclusions. | For any analytical claim about causation, ask: what confounders could explain this correlation? Is this an experiment (randomized) or an observational study? What assumptions are required to make the causal interpretation valid? Read Cunningham’s causal inference text before making causal claims from observational data. |
| Underpowered experiments | A/B tests with insufficient sample size fail to detect real effects, wasting the experimental budget, or detect noise, producing false positives. Sample size calculation is not optional — it is what determines whether an experiment can answer the question it is designed to answer. The temptation to “check early” and stop when results look good is a systematic bias toward false positives. | Calculate the required sample size before starting any experiment, based on the minimum detectable effect, significance level, and desired power. Pre-register the analysis plan and stopping rules. Do not look at results more than a few pre-specified times during data collection. The discipline is inconvenient; the alternative is unreliable results. |
| Tool identity over engineering fundamentals | The data engineering ecosystem changes rapidly: the dominant tools of 2015 (Hadoop, Hive) were largely displaced by 2020 (Spark, cloud warehouses); the tools of 2020 are being displaced by 2025. Practitioners who invest heavily in tool-specific expertise without understanding the underlying principles — distributed systems, SQL fundamentals, data modeling — must constantly re-learn as tools change. | Build tool expertise on a foundation of concepts: understand why Spark uses in-memory computation (to avoid the disk I/O cost that made MapReduce slow), why SQL is declarative (to allow optimizer flexibility), why data warehouses partition by date (to prune partitions during time-range queries). With the concepts, learning new tools is adaptation; without them, it is repetition. |
| Skipping the statistical foundations | Data science tools — pandas, scikit-learn, Spark MLlib — make it easy to run statistical analyses without understanding what they do. A regression that reports significant coefficients can be entirely misleading if its assumptions (linearity, homoscedasticity, independence of errors) are violated. A classifier that achieves 99% accuracy on a dataset where 99% of samples are the majority class has learned nothing. These problems are invisible without statistical understanding. | Study probability and statistics (§2.5) before studying data science tools. Understand what a p-value is before computing one. Understand what accuracy measures before optimizing it. The tools are fast to learn; the statistical understanding takes longer and is more valuable. |
| Treating the modern data stack as solved | The current dominant stack — cloud data warehouse, dbt, orchestration tool, feature store — is better than what it replaced, but it has its own failure modes. Data quality at the source is not solved by any downstream tool. The lineage of custom Python transformations is not tracked by dbt. The cost of cloud compute for large-scale transformations is substantial and often underestimated. The operational complexity of streaming pipelines remains high. | Engage with the limitations of current tools as seriously as with their capabilities. Read postmortems of data platform outages and data quality incidents. Understand what guarantees your pipeline infrastructure actually provides and what it does not. The right posture is informed confidence — knowing what the tools can and cannot do. |
中文
数据并不是事实；它是被记录下来的观察，而这意味着所有现实中的混乱都会随之而来。真实数据会迟到，会重复到达，会带着错误值到达，会以不符合约定的格式到达，会来自写入过程中崩溃的系统，有时甚至根本不会到达。数据工程是一门构建基础设施的学科，它把数据从产生之处移动到可被使用之处——可靠地、大规模地、带有可记录的血缘，并带有下游消费者所需要的质量保证。数据科学则是从数据中提取理解的学科：提出数据能够回答的问题，设计不会被噪声或偏差欺骗的分析，并以能够区分“数据支持什么”和“数据不支持什么”的方式传达结论。
这两门学科深度耦合。一个无法信任输入分析流水线的数据科学家，是在沙地上建房；一个不了解数据将被如何使用的数据工程师，则会构建出技术上正确、但分析上无用的基础设施。最清楚的边界表达是：数据工程问的是“我们如何在正确时间，把干净、可靠的数据送到正确地方？”；数据科学问的是“数据告诉了我们什么，我们对此有多确定？” 在数据产生价值之前，这两个问题都必须得到回答。
本节位于数据库（§4.4）与机器学习（§5.2）之间。数据库提供存储和查询基础设施；本节处理的是数据进入数据库之前发生的事情（摄取、转换、质量），可以用数据做什么分析（探索性分析、A/B 测试、因果推断），以及如何为建模准备数据。机器学习提供预测建模方法；本节处理的是特征工程、实验追踪和数据基础设施，这些内容使这些方法能够在规模化实践中真正可用。
前置知识：数据库（§4.4）——SQL 和基础关系概念会贯穿使用。概率与统计（§2.5）——分布、置信区间、假设检验，以及因果与相关的区别，是分析基础。算法（§2.6）——理解成本模型和数据结构权衡，是推理流水线性能的必要条件。
统计学如何遇到计算，以及碰撞产生了什么
数据科学和数据工程的历史，是两种在二十世纪大部分时间里独立发展的传统之间的碰撞：统计理论与计算基础设施。
统计理论成长于 Fisher、Neyman、Pearson 及其继承者在二十世纪上半叶的工作，主要处理通过有意实验或细致调查收集的小数据集。实践方式是手工计算，最多也只是机械计算器；一个有数百个观察值的数据集已经算大。其智识框架精确而严格：假设检验、置信区间、实验设计、方差分析。假设是显式的，方法会根据这些假设被验证。规模仍然在人类可以管理的范围内。
当计算机使更大数据集变得可处理时，计算统计学出现了。John Tukey 1977 年的 Exploratory Data Analysis 提出了一种实践和一种哲学：在检验假设之前，先看数据。画出分布。找出异常值。寻找你没有预料到的结构。Tukey 认为，假设检验框架虽然有价值，但在许多科学实践中过早被使用；更合适的第一步是探索，因为探索可能揭示假设本身被错误设定、数据收集存在缺陷，或真正有趣的问题与最初提出的问题不同。EDA 至今仍是任何数据分析的正确起点，而它在数据科学实践中的中心地位可以追溯到 Tukey。
把统计计算带给实践者的语言是 S。它由 John Chambers 及其同事于 1976 年起在 Bell Labs 开发。S 提供了一个交互式数据分析环境，把向量和矩阵作为一等对象，提供用于统计模型的公式记法，以及用于可视化的图形系统。S 后来被商业化为 S-Plus；它的开源重实现 R，由 Ross Ihaka 和 Robert Gentleman 于 1990 年代早期在 University of Auckland 开始开发，后来成为学术界和许多工业场景中统计计算的主导平台。R 的包生态系统 CRAN 到 2020 年已经积累了超过 15,000 个包，覆盖几乎所有统计方法，代表了科学领域中最有野心的协作软件项目之一。
Python 更逐步地进入数据分析领域：NumPy（2006）提供高效数组操作；pandas（2008）提供带标签的数据框，其设计大致借鉴了 R 的 data frame 概念；matplotlib（2003）用于可视化。这一组合——再加上用于机器学习的 scikit-learn，以及用于交互式计算的 Jupyter notebooks——使 Python 在 2010 年代早期成为数据科学的主导语言，尤其是在工业界，因为与生产软件系统集成的能力，比统计包生态的深度更重要。
计算规模问题随着 Web 到来。到 2000 年代早期，Google 和 Yahoo 等公司积累的数据量——日志、用户行为、爬取结果——已经无法在单台机器上处理。Google 2004 年的论文 “MapReduce: Simplified Data Processing on Large Clusters” 描述了一种用于分布式数据处理的编程模型：用户定义的 Map 函数转换每条记录；shuffle 阶段按 key 对转换后的记录分组；用户定义的 Reduce 函数聚合每个组。框架会自动处理分发、容错和并行。MapReduce 优雅而强大，Apache Hadoop 项目于 2006 年把它实现为开源软件。Hadoop 与 HDFS（Hadoop Distributed File System）结合，成为大规模数据处理的主导平台，并创造了由工具、公司和实践组成的“大数据”生态系统。
Hadoop 有根本限制。MapReduce 要求在各阶段之间把中间结果写入磁盘，使迭代算法——机器学习和图处理所必需的算法——变得极慢。通过 Hive 在 Hadoop 上执行 SQL 查询，比关系数据库上的 SQL 慢数个数量级。直接使用 MapReduce API 编程冗长且容易出错。Apache Spark 由 UC Berkeley 的 AMPLab 开发，并于 2012 年开源（2014 年公开发布），它通过内存计算和更高级的 API 解决了这些限制。Spark 的 RDD（Resilient Distributed Dataset）以及后来的 DataFrame 和 Dataset API，使能够用 Python、Scala 或 Java 表达转换的实践者，可以在不直接编写 MapReduce 的情况下进行分布式数据处理。Spark 成为大规模数据工程中主导性的批处理引擎，并且至今仍是生态系统的核心。
流式数据处理作为一个独立问题出现：并不是所有分析都能等待一个批处理任务完成。监控仪表盘、欺诈检测、实时推荐和许多其他应用，都要求在事件发生后的几秒甚至几毫秒内处理事件。Apache Kafka 由 LinkedIn 开发，并于 2011 年开源，它提供了一个分布式日志，可以以高吞吐量存储和服务事件流。Apache Flink（2014）以及更早的 Apache Storm 提供了流处理引擎。其架构模式——一个持久日志（Kafka）同时供给流处理器（用于实时分析）和批处理器（用于历史分析）——被称为 Lambda 架构，尽管其复杂性（维护两套分离代码路径）推动了后来的 Kappa 架构（把批处理视为流处理的特例）。
现代数据栈来自对 Hadoop 时代基础设施复杂性的不满。云数据仓库——Snowflake（2012）、Google BigQuery（2010）、Amazon Redshift（2012）——证明，如果计算可以弹性扩展、存储足够便宜，那么大规模 SQL 分析可以比 Hadoop 简单得多。dbt（data build tool，2016）把软件工程实践应用到数据转换中：把 SQL 转换作为版本控制下的代码，配套自动化测试、文档和血缘追踪。Airflow（2014）、Prefect 和 Dagster 为数据流水线提供工作流编排。这一组合——云数据仓库、转换工具和编排工具——成为“现代数据栈”，并在 2020 年代早期取代了工业界大多数用于分析工作负载的 Hadoop 部署。
最新的发展是数据工程与机器学习的融合。特征库（Feast、Hopsworks、Tecton）解决的是 ML 特有的数据工程问题：模型训练和推理所需的特征必须被一致地计算、高效地存储、低延迟地服务，并被追踪以保证可复现性。ML 平台（MLflow、Kubeflow、Vertex AI）增加了实验追踪、模型版本管理和部署基础设施。“MLOps” 运动把 DevOps 实践应用到机器学习中：持续训练流水线、自动化评估、部署自动化、对数据漂移和模型退化的监控。这些发展使数据工程与 ML 基础设施越来越整合，也要求实践者同时理解二者。
流水线、质量与分析栈
数据流水线：从来源到去向
数据流水线是一系列转换，它把数据从来源——生产数据库、事件流、第三方 API、文件上传——移动到去向——数据仓库、特征库、下游应用。流水线的责任不只是移动数据，还要维持数据正确性：确保每条记录只被处理一次，故障能够被优雅处理，操作具有幂等性（可以重试而不会产生重复结果），源系统中的 schema 变化不会破坏下游消费者，并且数据以消费者可以使用的形式到达。
在大多数现代分析架构中，ELT（Extract, Load, Transform）模式已经取代了 ETL（Extract, Transform, Load）。在 ETL 中，数据会在加载进数据仓库之前被转换，这要求在仓库之外维护复杂转换逻辑。在 ELT 中，原始数据先被加载进仓库，再使用 SQL 在仓库中转换。这一转变反映了云存储和云计算的经济性：存储已经足够便宜，可以保存原始数据；计算已经足够弹性，可以在 SQL 中转换大规模数据，而不需要单独的处理基础设施。主导性转换工具 dbt 实现的就是 ELT：它以 SQL SELECT 语句作为输入，并把它们物化为仓库中的表或视图，同时提供依赖追踪、测试和文档。
流水线可靠性要求认真思考失败模式：如果来源不可用会怎样？如果一条记录验证失败会怎样？如果下游消费者速度太慢会怎样？幂等性要求——重新运行一条流水线所产生的结果应与运行一次相同——是最重要的单一设计约束。非幂等流水线在重试时会产生重复数据，而一旦有任何环节失败，重试就是必要的。使操作具备幂等性通常需要使用 upserts（insert-or-update）而不是 inserts，追踪 watermarks 以知道哪些记录已经被处理，并使用相对于目标系统是原子的事务性加载模式。
数据编排工具——Apache Airflow、Prefect、Dagster——负责调度和协调流水线执行。任务的 DAG（有向无环图）规定依赖关系：任务 B 在任务 A 之后运行，任务 C 在 B 和 D 都完成之后运行。编排器处理失败重试，监控任务完成情况，在流水线迟到时发送警报，并提供历史运行的可见性。选择编排器时，需要在运维复杂度（Airflow 强大但运维要求高）、开发者体验（Prefect 和 Dagster 强调可用性），以及与更广泛数据栈的集成之间权衡。
数据质量：衡量和维护信任
数据质量不是二元属性。数据不是干净或肮脏；它在多个维度上具有可测量的质量——完整性（所有预期记录是否存在？）、新鲜度（数据有多近？）、准确性（值是否匹配其真实来源？）、一致性（相关值在不同表之间是否一致？）、唯一性（是否存在意外重复？）和有效性（值是否符合预期格式和范围？）。每个维度都可以被测量、追踪趋势并设置告警。
由 dbt 和 Great Expectations 等工具推广的数据质量测试，把软件测试实践应用到数据上。测试会规定对数据的期望：这一列不应为空，这个外键应总是匹配父表，这个指标应处于上周数值的 20% 范围内。测试会在数据加载或转换时自动运行，在质量问题传播到下游分析之前捕捉它们。过去十年中，从临时质量检查转向系统性自动化测试，是数据工程实践中最重要的改进之一。
数据血缘——追踪一个下游表来自哪些上游表，以及应用了哪些转换——对于调试质量问题至关重要。当一个下游指标意外变化时，血缘允许工程师把变化追溯到来源：哪个上游表变了？对它应用了哪种转换？哪个源系统是根本原因？现代转换工具（dbt）和元数据平台（DataHub、OpenMetadata、Atlan）会为 SQL 定义的转换自动维护血缘，尽管自定义代码的血缘仍然更难捕捉。
数据契约是一种较新的实践，它规定数据生产者与消费者之间的接口：生产者会提供哪些字段，它们的类型是什么，质量保证是什么，以及在变更前会提供什么通知。契约把过去只存在于文档或工程师脑中的隐含约定形式化，使它们可以被测试和自动化。数据契约模式处理的是数据工程中的根本摩擦：数据生产者（产品团队、运营系统）会优先优化自身需求，而当生产者行为在没有通知的情况下变化时，数据消费者（分析师、数据科学家、下游应用）就会被破坏。
分析方法：从探索到因果
探索性数据分析是任何严肃数据分析的第一步。在检验假设或拟合模型之前，分析者会检查数据：每个变量的分布是什么？异常值有哪些，应该如何处理？缺失值有哪些，它们意味着什么？变量之间存在哪些相关性？探索经常会揭示问题——数据收集错误、意外季节性、总体异质性——这些问题会使原本计划的分析失效，并把工作转向数据真正能够回答的问题。
A/B 测试（随机对照实验）是测量数字系统中因果效应的黄金标准。用户被随机分配到对照组和处理组；两组都会测量一个感兴趣指标；统计检验判断观察到的差异更可能来自处理，还是来自偶然。随机化确保两组之间唯一系统性差异是处理本身，从而使因果推断有效。不过，A/B 测试有要求：处理必须能在随机化单位上被分配，处理条件和对照条件必须定义清楚，指标必须能够在不受组间污染的情况下测量，并且需要足够样本量来检测预期效应。许多商业问题无法通过 A/B 测试回答，例如历史比较、网络效应，以及随机化的伦理限制，因此需要替代方法。
没有随机化的因果推断——使用观察数据估计因果效应——是数据科学中最困难的方法论问题。根本挑战是混杂：影响处理和结果的因素会制造出看似因果的关系，即使真正的因果关系并不存在。双重差分、工具变量、回归不连续和合成控制，是在无法随机化时处理特定混杂结构的标准方法。每种方法都需要由领域知识证明其假设合理，并测试这些假设是否可信；没有任何方法可以普遍适用。一个把观察数据中的相关关系直接当成因果关系、而不处理混杂的实践者，会系统性地产生错误结论。
实验设计不是事后问题。在收集数据之前，就必须根据给定效应大小、显著性水平和统计功效，计算检测该效应所需样本量。功效不足的研究无法检测真实效应；如果研究者在结果尚不显著时查看数据并决定继续收集，则会抬高第一类错误率。预注册——在看到数据之前明确假设、设计、分析计划和停止规则——是区分验证性分析和探索性分析的实践。社会科学、心理学和医学中的可复现性危机，已经大规模展示了统计学家一直知道的事实：功效不足的研究，加上灵活的分析决策，会产生无法复现的发现。
学习这一部分会改变什么
数据工程和数据科学会改变实践者思考主张背后证据的方式，也会改变他们理解生成这些证据的基础设施的方式。
第一个变化，是流水线思维：本能地把任何数据产品追溯到其来源，并质疑流水线是否可靠。一个看起来不错的指标，如果是由迟到数据、错误去重的数据，或带有转换 bug 的数据计算出来的，就毫无意义。学过数据工程的实践者看到任何仪表盘或分析，都会立刻问：这些数据来自哪里？它们如何被转换？如果它们错了，我如何知道？这种本能可以防止一种常见失败：基于看似权威、实际损坏的数据做决策。
第二个变化，是统计纪律：习惯于区分数据展示了什么，以及数据证明了什么。相关不意味着因果。统计显著不意味着实践显著。一个在 n = 10,000 个观察值下显著的结果，可能没有实际意义——效应真实但极小。一个在 n = 100 个观察值下不显著的结果，几乎说明不了什么。学过数据科学的实践者会自动做出这些区分，并对没有这样区分的主张保持适当怀疑。
第三个变化，是规模直觉：在构建之前预测某种数据处理方法会产生什么成本的能力。处理 100 行和处理 1 亿行需要不同方法。一个在 SQL 中很简单的 join，在分布式系统中会变成跨网络移动 TB 级数据的 shuffle 操作。一个在本地机器上需要毫秒的聚合，如果数据没有分区并分散在许多文件中，可能需要数小时。这种成本直觉会指导架构选择，并防止那种在开发环境中可行、在生产环境中失败的常见系统建设错误。
第四个变化，是正确设计实验并解释其结果的能力。并不是每个关于产品表现、用户行为或商业指标的主张，都有良好设计的实验支持。理解实验设计的实践者，能够识别一个被声称的因果关系什么时候可能受到混杂影响，样本量什么时候不足以检测有意义的效应，以及看到数据之后做出的分析选择什么时候已经使统计检验失效。随着组织越来越多地根据数据做决策，这种能力变得越来越有价值。
资源
书籍与文本
Kleppmann 的 Designing Data-Intensive Applications（O’Reilly，2017，已在 §4.4 引用）深入覆盖数据工程背后的分布式系统和数据库基础。对于数据工程师来说，关于批处理、流处理和分布式数据系统的章节尤其相关。它是理解数据流水线所运行基础设施的最佳单卷本。
Reis 和 Housley 的 Fundamentals of Data Engineering（O’Reilly，2022）是把数据工程作为一门学科来处理的最全面文本：数据工程生命周期（生成、摄取、转换、服务）、每个阶段的工具和平台，以及在它们之间做选择的决策框架。它比 Kleppmann 更具规定性，概念深度略弱，但对完整生态的覆盖更全面。
对于数据分析方法论，Tukey 的 Exploratory Data Analysis（Addison-Wesley，1977）仍是基础文本。它在计算工具上已经过时，但在哲学上没有过时：在检验假设之前先看数据，对单一数字摘要保持怀疑，先理解结构再拟合模型。任何严肃数据实践者都应当读它。
Gelman 和 Hill 的 Data Analysis Using Regression and Multilevel/Hierarchical Models（Cambridge，2007）是面向数据科学家的应用统计建模最佳文本。它覆盖回归、把回归推广到层级数据结构，以及模型检查和验证，并采用贝叶斯视角，使不确定性量化变得自然。书中实践例子来自真实社会科学研究，使方法变得具体。
Cunningham 的 Causal Inference: The Mixtape（Yale University Press，2021，可在 mixtape.scunning.com 免费在线获取）覆盖了使用观察数据进行因果推断的主要方法——潜在结果框架、双重差分、工具变量、回归不连续、合成控制——并提供 Stata 和 R 的完整案例。对于有基础统计背景的实践者来说，这是进入因果推断文献最易读的入口。
Imbens 和 Rubin 的 Causal Inference for Statistics, Social, and Biomedical Sciences（Cambridge，2015）提供了因果推断的正式统计处理。它比 Cunningham 更难，但也更严格；适合那些不仅想知道如何应用方法，也想理解方法为什么有效的实践者。
对于 ML 数据基础设施侧，Huyen 的 Designing Machine Learning Systems（O’Reilly，2022）以系统视角覆盖特征工程、训练数据、数据和特征流水线，以及 ML 部署。对于构建 ML 基础设施的实践者来说，这是最有用的单本书。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Kleppmann, Designing Data-Intensive Applications | 数据工程的分布式基础 | 深入 |
| Reis & Housley, Fundamentals of Data Engineering | 综合数据工程生态 | 入门 |
| Tukey, Exploratory Data Analysis | 基础数据分析哲学 | 入门 |
| Gelman & Hill, Data Analysis Using Regression and Multilevel Models | 应用统计建模 | 深入 |
| Cunningham, Causal Inference: The Mixtape（免费） | 面向实践者的因果推断方法 | 深入 |
| Imbens & Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences | 正式因果推断 | 深入 |
| Huyen, Designing Machine Learning Systems | ML 基础设施与数据流水线 | 深入 |
课程与讲座
Stanford CS 246（Mining Massive Datasets，材料和视频免费在线获取）覆盖大规模数据处理的算法基础：局部敏感哈希、降维、PageRank、推荐系统和分布式数据挖掘算法。它连接了算法理论与大规模处理数据的实践挑战。
The Missing Semester of Your CS Education（MIT，免费，missing.csail.mit.edu）覆盖实践计算技能——shell 脚本、版本控制、使用命令行工具进行数据整理——这些是数据工程师的日常工具箱，但很少在 CS 课程中被教授。其中关于 data wrangling 的讲座（使用 sed、awk、grep 和相关工具）尤其有价值。
Kaggle Learn（免费）提供关于 pandas、SQL、特征工程和数据可视化的结构化 notebooks。这些教程实践性强、交互式，适合作为深入概念学习之前的工具入门。
dbt Learn（免费，courses.getdbt.com）提供 dbt 基础和高级功能的结构化训练。考虑到 dbt 在现代数据栈中的中心地位，任何使用数据仓库工作的实践者都值得完成它。
| 课程 | 平台 | 类型 |
|---|---|---|
| Stanford CS 246 Mining Massive Datasets（免费） | Stanford / YouTube | 深入 |
| MIT Missing Semester（免费） | missing.csail.mit.edu | 入门 |
| dbt Learn（免费） | courses.getdbt.com | 实践 |
| Kaggle Learn（免费） | kaggle.com/learn | 入门 |
| DataTalksClub Data Engineering Zoomcamp（免费） | DataTalksClub / GitHub / YouTube | 实践 |
| Dagster University and documentation（免费） | dagster.io / docs.dagster.io | 实践 |
实践、工具与当前资料
数据工程工具箱主要由工具而不是可视化组成。需要学习的核心工具是：
数据仓库上的 SQL 是最重要的技能。在 Snowflake、BigQuery 或 DuckDB 中针对真实数据集编写 SQL；使用 EXPLAIN 查看查询计划；观察不同 join 顺序、有无 clustering 或 partitioning 时查询性能如何变化。DuckDB（免费，本地运行）是最好的学习环境：它可以高速处理本地 CSV 和 Parquet 文件，并提供完整 SQL 实现，包括窗口函数、lateral joins 和其他高级功能。
dbt 与本地或云数据仓库 构成转换层。dbt 教程（可在 docs.getdbt.com/docs/get-started/dbt-core-quickstart 免费获取）会带你用示例数据集创建 models、tests 和 documentation。在教程基础上扩展自定义 models 和 tests，是合适的后续练习。
通过 PySpark 使用 Apache Spark 来进行大规模批处理。Databricks Community Edition（免费）提供带 notebooks 的托管 Spark 环境。处理一个大到本地 pandas 无法处理的数据集——例如数亿行——并观察不同分区策略、join 算法和聚合方法之间的性能差异，可以建立规模直觉。
Apache Kafka 用于事件流。Confluent 的免费层和教程提供托管 Kafka 环境。完成一个流式数据流水线教程——产生事件、消费事件、应用转换——可以使流处理范式变得具体。
Jupyter notebooks 配合 pandas、matplotlib 和 seaborn 用于探索性数据分析。最有教育价值的练习，是拿一个真实数据集，在没有预设假设的情况下做 EDA，记录发现，然后基于探索设计分析。UCI Machine Learning Repository 和 Kaggle datasets 都提供了合适起点。
| 资源 | 平台 | 类型 |
|---|---|---|
| DuckDB（免费） | duckdb.org | 实践 |
| dbt Core（免费） | dbt-core / docs.getdbt.com | 实践 |
| Great Expectations（免费 / 付费 cloud） | docs.greatexpectations.io | 实践 |
| Apache Arrow columnar format（免费） | arrow.apache.org | 参考 |
| Apache Parquet documentation（免费） | parquet.apache.org | 参考 |
| Polars user guide（免费） | docs.pola.rs | 实践 |
| DataHub metadata platform（免费 / 付费 cloud） | docs.datahub.com | 参考 |
| PySpark on Databricks Community Edition（免费） | databricks.com | 实践 |
| Apache Kafka / Confluent free tier | confluent.io | 实践 |
| Jupyter + pandas + matplotlib（免费） | jupyter.org | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 不验证就信任流水线 | 一个看起来合理、但由损坏数据计算出的指标，会以高度自信的形式产生错误结论。数据质量问题普遍存在，而且经常不可见：迟到数据看起来像记录缺失，去重 bug 导致事件被重复计数，时区处理错误让日期偏移一天，schema 变化悄悄破坏下游转换。不验证数据质量的实践者，是在不稳定地基上做分析。 | 在把任何数据集用于重要决策之前，追踪产生它的流水线：来源是什么？应用了哪些转换？重复可能在哪里进入？是否有自动质量测试？在分析前对每个数据集运行基础 sanity checks：按日期统计行数、关键列空值率、数值范围检查。把数据质量视为每次分析的第一步。 |
| 混淆相关与因果 | 观察数据展示的是相关；确立因果需要额外假设或实验设计。那些在转化前花了更多时间浏览网站的用户，不一定是因为花了更多时间才转化——本来就会转化的用户可能浏览得更多。混杂在行为数据中无处不在，把观察相关当成因果关系的分析会系统性地产生错误结论。 | 对任何关于因果的分析主张，都要问：哪些混杂因素可能解释这个相关？这是实验（随机化）还是观察研究？要让因果解释成立，需要哪些假设？在根据观察数据提出因果主张之前，阅读 Cunningham 的因果推断教材。 |
| 功效不足的实验 | 样本量不足的 A/B 测试无法检测真实效应，浪费实验预算，或者检测到噪声，产生假阳性。样本量计算不是可选项——它决定一个实验能否回答自己被设计来回答的问题。想要“提前看看”，并在结果看起来不错时停止，是一种系统性偏向假阳性的做法。 | 在开始任何实验之前，根据最小可检测效应、显著性水平和期望功效，计算所需样本量。预注册分析计划和停止规则。数据收集过程中，不要查看结果超过少数几个预先规定的时间点。这种纪律不方便；但替代方案是不可靠结果。 |
| 工具身份高于工程基础 | 数据工程生态变化很快：2015 年的主导工具（Hadoop、Hive）到 2020 年已经很大程度上被替代（Spark、云数据仓库）；2020 年的工具也正在被 2025 年的工具替代。只重度投资工具特定经验，而不理解底层原则——分布式系统、SQL 基础、数据建模——的实践者，会随着工具变化不断重新学习。 | 把工具能力建立在概念基础上：理解 Spark 为什么使用内存计算（为了避免让 MapReduce 变慢的磁盘 I/O 成本），理解 SQL 为什么是声明式语言（为了让优化器有灵活性），理解数据仓库为什么按日期分区（为了在时间范围查询中裁剪分区）。有概念，学习新工具就是适配；没有概念，就是重复。 |
| 跳过统计基础 | 数据科学工具——pandas、scikit-learn、Spark MLlib——让人在不了解其原理的情况下也能运行统计分析。一个报告显著系数的回归，如果违反了其假设（线性、同方差性、误差独立性），可能完全误导。一个在 99% 样本属于多数类的数据集上达到 99% 准确率的分类器，其实什么也没学到。没有统计理解，这些问题是不可见的。 | 在学习数据科学工具之前，先学习概率与统计（§2.5）。在计算 p-value 之前，先理解 p-value 是什么。在优化 accuracy 之前，先理解 accuracy 衡量什么。工具很快能学会；统计理解更慢，但更有价值。 |
| 把现代数据栈当成已经解决的问题 | 当前主导栈——云数据仓库、dbt、编排工具、特征库——比它所取代的东西更好，但它也有自己的失败模式。任何下游工具都解决不了源头数据质量问题。自定义 Python 转换的血缘不会被 dbt 追踪。大规模转换的云计算成本很高，而且经常被低估。流式流水线的运维复杂性仍然很高。 | 像认真理解当前工具能力一样，认真理解它们的限制。阅读数据平台故障和数据质量事故的事后复盘。理解你的流水线基础设施实际提供了什么保证，又没有提供什么保证。正确姿态是有根据的信心——知道工具能做什么，也知道它们不能做什么。 |