English
Testing is the activity of executing software to check whether it behaves correctly — where “correctly” means conforming to some specification, formal or informal. This definition contains a significant limitation: testing can only demonstrate the presence of bugs, never their absence. Edsger Dijkstra stated this in 1972, and the logic is straightforward. A program’s correctness for all inputs requires proof; a test demonstrates correctness for the specific inputs tested. Unless the input space is finite and small enough to exhaust completely, testing is sampling, and sampling has inherent coverage gaps. The question is not whether to test — testing is necessary — but how to test well enough that the coverage gaps are manageable and not catastrophic.
The discipline of software testing provides structured answers to that question: which test types to write, in what proportions, at what level of abstraction, with what tools, using what strategies for generating inputs, and how to integrate testing into the development process so that it provides continuous feedback rather than a one-time gate. The answers have changed substantially over the past twenty years, driven by the development of automated testing infrastructure, test-driven development as a practice, continuous integration as a deployment model, and property-based testing as a method for generating inputs the developer would not have thought of.
Quality assurance is the broader discipline of which testing is one part. QA includes defining quality attributes — correctness, reliability, performance, security, maintainability, usability — and establishing the practices and processes that produce software with those attributes. Testing provides evidence about correctness and the presence or absence of known defect types; QA provides the framework for asking whether the evidence is sufficient, what other quality attributes need attention, and how the development process should be structured to prevent defects rather than only detect them.
Prerequisites: Programming (§2.1) — writing tests is programming. Algorithms (§2.6) — understanding computational complexity helps reason about test coverage. Software architecture (§7.1) — testability is an architectural property that must be designed in.
From Ad Hoc Verification to Test-Driven Development
Software has been tested since there was software to test, but the systematic study of testing as a discipline began in the late 1970s.
The earliest software testing was exploratory: a programmer who wrote a program would run it on some inputs and check whether the output looked right. This was not negligent — it was the only available option. The tools for systematic testing did not exist, the theory of test coverage had not been developed, and the concept of automated regression testing was not yet articulated. The 1950s and 1960s produced the first generation of programmers who knew that debugging was expensive and that finding bugs late in development was much more expensive than finding them early, but had no systematic framework for doing so.
Gerald Meyers’s The Art of Software Testing (1979) was the first systematic treatment of software testing as an engineering discipline. Meyers articulated the conceptual distinction between black-box testing (testing a component through its external interface, without knowledge of its implementation) and white-box testing (testing with knowledge of the implementation, to cover specific code paths). He introduced equivalence partitioning — dividing the input space into classes of inputs that should produce the same result, then testing one representative from each class — as a method for choosing which inputs to test systematically. He articulated boundary value analysis — testing at the edges of input ranges, where bugs most commonly occur — as a complementary technique. These concepts, developed for manual test case design, remain foundational to systematic test design half a century later.
The concept of code coverage — measuring what fraction of the code’s branches, statements, or paths are exercised by a test suite — emerged in the 1970s as a metric for testing adequacy. Branch coverage (also called decision coverage) measures whether both true and false branches of each conditional have been executed by at least one test. This provided a computable proxy for thoroughness, even if the proxy was imperfect. A test suite with 100% branch coverage might still miss bugs — coverage measures what code was executed, not whether the executed code was tested for the right behavior.
The software crisis of the 1970s and 1980s — large software projects consistently delivered late, over budget, and with significant numbers of defects — motivated the software engineering discipline to take testing more seriously. The V-model of development, which paired each development phase with a corresponding testing phase (unit testing corresponding to detailed design, integration testing to high-level design, system testing to requirements), formalized the relationship between testing and development phases. This model was adopted in government and aerospace standards and shaped how large organizations structured testing for decades.
The emergence of object-oriented programming in the late 1980s and 1990s brought new testing challenges. Object-oriented code was harder to test in isolation than procedural code because objects had state that accumulated through sequences of method calls. The concepts of setup (establishing the state before a test), exercise (executing the behavior under test), and verify (checking that the result is correct) became the standard test structure. Kent Beck, working at Smalltalk in the early 1990s, developed SUnit as the first unit testing framework implementing this structure; JUnit, which Beck and Erich Gamma developed for Java in the mid-1990s, brought the xUnit pattern to a much wider audience. JUnit popularized the concept of running all tests automatically as part of the build, making automated regression testing practical.
Martin Fowler and Kent Beck’s articulation of continuous integration — the practice of integrating code changes into a shared repository multiple times per day, with an automated build that includes tests — formalized the connection between automated testing and software development practice. Fowler’s 2000 article “Continuous Integration” (later expanded in 2006) described the practice and its benefits: because integration happens frequently, integration problems are discovered immediately rather than accumulating over weeks of parallel development. The automated test suite that ran on each integration provided the signal that integration had not broken existing behavior. Continuous integration made the feedback loop from code to test result fast enough that testing became a natural part of development rather than a phase at the end.
Test-Driven Development (TDD), formalized by Kent Beck in Test-Driven Development: By Example (2002), inverted the conventional relationship between writing code and writing tests. In TDD, the developer writes a test for the desired behavior before writing the code that implements it, watches the test fail (verifying the test actually tests something), writes the minimum code to make the test pass, and then refactors the code while keeping tests passing. The cycle — Red, Green, Refactor — produces code that is tested from birth, that has the testability built in (since it was written to be tested), and that tends toward minimal implementation because only enough code to pass the current tests is written. TDD has empirical support for reducing defect rates and has influenced how a generation of developers approaches design; it has also been criticized for being impractical in domains where the expected behavior is not known upfront and for encouraging over-reliance on unit tests at the expense of integration and end-to-end tests.
Property-based testing, introduced by Koen Claessen and John Hughes in their 2000 paper “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs,” provided a different answer to the question of which inputs to test. Instead of specifying individual examples, the developer specifies properties — general statements about what the code should do — and the testing framework generates random inputs to find counterexamples. QuickCheck for Haskell spawned implementations in most major languages (Hypothesis for Python, fast-check for JavaScript, jqwik for Java). Property-based testing finds classes of bugs that example-based tests miss because it generates inputs the developer would not have thought of; its limitation is that specifying useful properties requires more abstract thinking than specifying examples.
The testing pyramid, described by Mike Cohn in Succeeding with Agile (2009) and elaborated by Fowler and others, proposed that test suites should have many unit tests, fewer integration tests, and even fewer end-to-end tests. The pyramid shape reflects the speed and cost tradeoffs: unit tests run in milliseconds and isolate failures precisely; end-to-end tests run in seconds or minutes and when they fail require substantial investigation to identify the cause. The pyramid as a prescription has been refined over time — the “testing trophy” (popularized by Kent C. Dodds) emphasizes integration tests more heavily — but the underlying principle, that different test types serve different purposes and should be used in proportions calibrated to those purposes, remains sound.
The current period is characterized by test infrastructure that previous generations would have found remarkable: continuous integration systems that run the full test suite on every commit, coverage measurement that tracks trends over time, mutation testing that evaluates test quality by injecting deliberate bugs and checking whether tests catch them, contract testing that validates assumptions between services, and observability infrastructure that makes production behavior measurable. AI assistance for test generation — suggesting test cases, writing test scaffolding, generating property specifications — is being integrated into development tools. The fundamental challenges have not changed: the input space is infinite, the oracle problem (knowing what the correct output should be) is hard, and writing tests that are maintainable, fast, and reliable remains a discipline requiring skill and judgment.
Test Types, Coverage, and Test Quality
The Test Hierarchy and What Each Level Provides
Unit tests verify the behavior of individual functions or classes in isolation from their dependencies. A function that computes the nth Fibonacci number, a class that validates email addresses, a parser that converts a string to a date — each can be tested independently, with inputs provided programmatically and outputs checked against expected values. Unit tests are fast (milliseconds), deterministic (same inputs produce same outputs), and isolating (a failing test points to a specific function or class). Their limitation is that they do not test the interactions between components; a system where every unit tests passes can still fail when the units are combined.
Test doubles — the general term encompassing stubs (which return predetermined values), mocks (which verify that certain calls were made), spies (which record calls), fakes (which implement a simplified version of a dependency), and dummies (which fill parameter lists but are never used) — allow unit tests to isolate the component under test from its dependencies. A function that sends email can be tested by providing a mock email sender that records calls rather than actually sending email. The risk is that test doubles that are poorly synchronized with the real components they replace pass their tests while the real system would fail; contract tests between components help address this.
Integration tests verify the behavior of components working together, often with real (not mocked) dependencies. A test that exercises a REST endpoint, sends a real HTTP request to a running service, and checks the HTTP response is an integration test. Integration tests are slower than unit tests (they may involve network calls, database queries, file operations) and are harder to isolate (a failure may involve any of the interacting components), but they verify behavior that unit tests cannot: that components correctly interpret each other’s data formats, handle each other’s error conditions, and agree on the semantics of shared interfaces.
End-to-end tests verify the behavior of the complete system through the user interface. A Playwright or Selenium test that logs in, navigates to a form, fills it out, submits it, and checks the resulting state is an end-to-end test. End-to-end tests provide the most realistic verification of user-visible behavior and the most expensive and flaky: they depend on browser behavior, timing, network conditions, and all the services the UI calls. A small suite of end-to-end tests that covers the most critical user journeys provides value; a large suite becomes a maintenance burden whose failures are hard to diagnose.
Contract testing, particularly consumer-driven contract testing as implemented by Pact, verifies the assumptions that one service makes about another’s API. The consumer defines the interactions it expects (request format, response format, status codes); the provider verifies that it can fulfill those interactions. This enables teams to test service boundaries without running all services simultaneously and catches API incompatibilities before they reach production.
Mutation testing evaluates test quality by introducing deliberate bugs (mutations) into the source code — changing a + to a -, a > to a >=, a true to a false — and checking whether the test suite catches the mutation (the mutation is “killed”) or misses it (the mutation “survives”). Surviving mutations indicate that the test suite does not adequately verify the behavior that the mutated code implements. Mutation testing is computationally expensive but provides a more meaningful quality metric than code coverage: coverage measures execution, mutation testing measures verification.
Test Coverage and Its Limitations
Coverage metrics — statement, branch, path, mutation — measure different aspects of how thoroughly a test suite exercises the code. Branch coverage (has each conditional’s true and false branch been exercised?) is the most commonly required metric and the most meaningful of the simple metrics. 100% branch coverage is achievable on most code and is a reasonable minimum standard for critical code; it does not guarantee the absence of bugs.
Coverage is a lower bound on test quality, not an upper bound. A test that exercises every branch but does not assert meaningful outputs (or asserts only that no exception was thrown) has full coverage and tests nothing. Coverage tools can be gamed: a developer who wants to achieve a coverage target can write tests that execute code without asserting anything. The meaningful standard is not “what code has been executed” but “what behaviors have been verified.”
The oracle problem — the question of what the correct output should be — is where test quality ultimately lives. For many functions, the correct output is easy to specify: the sum of two integers, the sorted order of a list. For others, it requires domain knowledge that may be difficult to encode: whether a classified image is correct, whether a generated text is acceptable, whether a recommendation is relevant. Property-based testing addresses oracle limitations partially by specifying invariant properties (the output is a permutation of the input, the result is non-negative) rather than exact outputs, but even properties require thoughtful specification.
Test Design Principles
Tests that are hard to maintain are tests that will be abandoned. The primary enemies of test maintainability are coupling to implementation rather than behavior (tests that break when the implementation changes but the behavior does not), unclear test structure (tests whose purpose is not obvious from their code), and insufficient isolation (tests that pass or fail depending on the execution order or on external state).
The Arrange-Act-Assert (AAA) pattern structures tests clearly: arrange the preconditions and inputs, act by executing the behavior under test, assert that the postconditions are correct. Each test should ideally test one behavior; a test that fails should identify a single specific problem, not any of several possible problems. Test names should describe the behavior being verified (“when_user_email_is_invalid_registration_should_fail”) rather than the mechanics being exercised (“test_registration_with_invalid_email”).
Tests that depend on shared mutable state are flaky: they pass or fail depending on which tests ran before them and in what order. Each test should set up the state it needs and tear down any state it creates. Database tests are particularly prone to this issue; running each test in a transaction that is rolled back at the end, or truncating and seeding the database before each test run, provides isolation.
What Studying This Changes
Testing changes how practitioners think about code during development rather than after it.
The first change is testability as a design constraint. Code that is hard to test is hard to change: the same coupling that makes a function difficult to test in isolation makes it difficult to modify without understanding and changing all the dependencies. Practitioners who think about testing during design write code with injected dependencies (rather than hard-coded ones), small functions with clear inputs and outputs, and well-defined boundaries between components. The result is code that is both more testable and more maintainable.
The second change is the ability to distinguish test types and use them appropriately. A developer who knows only that “tests go here” writes tests at whatever level is most convenient rather than most appropriate. A practitioner who understands the different purposes of unit, integration, and end-to-end tests can construct a test suite where each type provides specific value and the proportions are calibrated to the costs and benefits.
The third change is the discipline of treating failures seriously. A test suite that is partially ignored — “we have 500 failing tests, that’s just how it is” — provides no value and misleads: it implies the system is tested when it is not. Practitioners who take testing seriously keep the test suite green: every failing test is either fixed or explicitly marked as known-failing with a tracked issue. This discipline is harder to maintain than it sounds and is the difference between a testing culture and a testing checkbox.
The fourth change is thinking probabilistically about what kinds of bugs tests are likely to find. Example-based tests find bugs that the developer thought of; property-based tests find bugs in the space between examples; mutation testing reveals where the tests’ assertions are weaker than their coverage implies. A practitioner who understands the relationship between testing strategy and the kinds of bugs each strategy can and cannot find is better positioned to design a test suite appropriate to the system’s risk profile.
Resources
Books and Texts
Meyers, Sandler, and Badgett’s The Art of Software Testing (3rd ed., Wiley, 2012) is the foundational reference, updated from the 1979 original. Its systematic approach to test case design — equivalence partitioning, boundary value analysis, cause-effect graphing, error guessing — remains relevant to any testing effort. The black-box techniques it develops are independent of implementation language or framework.
Beck’s Test-Driven Development: By Example (Addison-Wesley, 2002) is the foundational TDD text. Its two worked examples (a money calculation in Java, a JUnit implementation in Python) demonstrate the TDD cycle in practice. The book’s influence on testing culture has been enormous; reading it provides direct access to the reasoning behind the practice.
Khorikov’s Unit Testing: Principles, Practices, and Patterns (Manning, 2020) is the most useful contemporary text on the principles of good unit testing. Its distinctions — between observing behavior and implementation details, between high-value tests and low-value tests, between sociable and solitary unit tests — provide a framework for evaluating test quality that goes beyond coverage metrics. The discussion of classical versus London schools of TDD (where to use mocks and where not to) is the clearest treatment of that controversy.
Freeman and Pryce’s Growing Object-Oriented Software, Guided by Tests (Addison-Wesley, 2009) demonstrates TDD at the integration level, starting from an end-to-end acceptance test and driving the design of components through the test-first discipline. Its outside-in approach, in contrast to Beck’s inside-out approach, addresses the limitation of starting from unit tests with mocks.
Humble and Farley’s Continuous Delivery (Addison-Wesley, 2010) situates testing in the deployment pipeline, covering the automated test stages (unit, integration, acceptance, performance), the conditions under which each stage runs, and the design of a pipeline that provides fast feedback on each commit.
| Book | Role | Type |
|---|---|---|
| Meyers, Sandler & Badgett, The Art of Software Testing (3rd ed.) | Foundational test design techniques | Depth |
| Beck, Test-Driven Development: By Example | TDD foundational text | Entry |
| Khorikov, Unit Testing: Principles, Practices, and Patterns | Contemporary unit testing principles | Practice |
| Freeman & Pryce, Growing Object-Oriented Software, Guided by Tests | TDD outside-in approach | Depth |
| Continuous Delivery testing chapters (§7.4 reference) | Testing in deployment pipeline | Depth |
| Osherove, The Art of Unit Testing (3rd ed.) | Practical unit testing reference | Entry |
Courses, Papers, and Current Sources
Dijkstra’s “Notes on Structured Programming” (1972, free online) contains the oft-quoted “program testing can be used to show the presence of bugs, but never to show their absence.” Reading the context — Dijkstra’s broader argument about the mathematical verification of programs — provides perspective on what testing can and cannot claim.
Fowler’s testing articles on martinfowler.com (free) — particularly “UnitTest,” “IntegrationTest,” “TestDouble,” “TestingStrategy,” and “Mocks Aren’t Stubs” — are the most careful contemporary treatments of testing concepts and their distinctions. The “Practical Test Pyramid” article by Ham Vocke (free on martinfowler.com) applies the pyramid to contemporary web application testing.
Claessen and Hughes’s “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs” (2000, free on ACM DL) is the paper that introduced property-based testing. Reading it provides the conceptual basis for the technique independent of the specific implementation.
| Resource | Platform | Type |
|---|---|---|
| Dijkstra, “Notes on Structured Programming” (1972, free) | Foundational limits of testing | Depth |
| Fowler, testing articles (free) | martinfowler.com | Reference |
| Claessen & Hughes, QuickCheck paper (free) | Property-based testing foundation | Depth |
| Google Testing Blog / Tech on the Toilet (free) | testing.googleblog.com | Auxiliary |
Practice, Tools, and Current Sources
Every major programming language has standard testing frameworks. JUnit 5 (Java), pytest (Python), Jest (JavaScript/TypeScript), RSpec (Ruby), Go testing package — the specific framework matters less than developing fluency with the framework used in your primary language, including parameterized tests, fixtures, and the test runner’s output.
Coverage tools — JaCoCo (Java), Coverage.py (Python), Istanbul/nyc (JavaScript) — measure which branches and statements are exercised. Integrating coverage measurement into CI and tracking coverage trends over time makes coverage a meaningful signal rather than a one-time assessment.
Hypothesis (Python, free) and fast-check (JavaScript, free) are mature property-based testing libraries. Writing property-based tests for a sorting function, a parser, or a data validation library — where the properties are clear — provides direct experience with how property-based testing finds bugs that example-based tests miss.
Pact (free, pact.io) implements consumer-driven contract testing across multiple languages. Working through the Pact example — defining a consumer contract, verifying a provider against it — makes the contract testing workflow concrete.
PIT (Java mutation testing, free) and Stryker (JavaScript/TypeScript mutation testing, free) run mutation testing against an existing test suite. Running mutation testing on a codebase you have written and examining the surviving mutants is the fastest way to discover where the test suite’s assertions are weaker than the coverage implies.
| Resource | Platform | Type |
|---|---|---|
| JUnit 5 / pytest / Jest (free) | Language-specific | Practice |
| Coverage tools: JaCoCo / Coverage.py / Istanbul (free) | Language-specific | Practice |
| Hypothesis (Python, free) | hypothesis.works | Practice |
| fast-check (JavaScript, free) | GitHub | Practice |
| Playwright Test (free) | playwright.dev | Practice |
| Pact contract testing (free) | pact.io | Practice |
| PIT mutation testing (Java, free) | pitest.org | Practice |
| Stryker (JS/TS mutation testing, free) | stryker-mutator.io | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Treating coverage as a quality proxy | 100% branch coverage is achievable with tests that assert nothing. A test suite can have high coverage and low quality: it exercises the code without verifying that the code does the right thing. Coverage is a necessary but not sufficient condition for test suite quality. | Evaluate test quality through mutation testing (do the tests catch deliberate bugs?) and through test review (does each test verify a behavior, or just execute code?). Set coverage thresholds as a floor, not a ceiling, and use mutation score alongside coverage to assess suite quality. |
| Over-mocking at the unit level | Tests that mock every dependency of a unit under test verify that the unit makes the calls it makes, not that the calls produce the right results. When the real dependencies are replaced entirely by mocks, integration problems are invisible to the test suite. The “London school” approach — test units in complete isolation with mocks — produces test suites that are brittle (mocks must be updated when any dependency changes) and that miss integration bugs. | Prefer integration tests where the interaction between components is being tested. Use mocks selectively, for dependencies that are slow, non-deterministic, or that have external side effects (email sending, payment processing). Fowler and Khorikov’s treatments of when to mock versus when to use real dependencies provide the framework for these decisions. |
| Tests that test implementation rather than behavior | A test that verifies that a function makes a specific sequence of internal method calls will break when the implementation is refactored, even if the visible behavior is unchanged. Such tests are a tax on refactoring: every internal change requires updating tests regardless of whether the behavior changed. | Write tests against the public interface, not the implementation. A test for a sorting function should verify that the output is sorted and is a permutation of the input; it should not verify that the function calls partition before calling quicksort. The implementation can change freely as long as the behavior (observable through the public interface) is preserved. |
| The test orphan problem | Tests that are written and then not maintained become increasingly misleading: they pass on code that has changed since they were written, or they fail for reasons unrelated to the behavior they test. A failing test suite that is mostly ignored (because “those tests always fail”) is worse than no test suite: it provides the organizational comfort of “we have tests” without the actual safety net. | Maintain a zero-tolerance policy for test failures: every test in the suite either passes or is explicitly marked as a known failure with a tracked issue. If a test is consistently failing for reasons that are not bugs, delete it or fix it. A small green test suite is more valuable than a large red one. |
| Testing only the happy path | Tests that cover only the expected, well-formed inputs and the successful execution paths miss the bugs that cause production failures: null inputs, empty lists, strings in unexpected encodings, concurrent access, network failures, database constraint violations. Production systems encounter the full range of inputs, not just the clean ones the developer had in mind. | Apply boundary value analysis and equivalence partitioning systematically: what happens with empty inputs? with inputs at the boundary of valid ranges? with inputs that combine multiple edge conditions? Use property-based testing to generate inputs the developer would not have considered. Review bug reports for the kinds of inputs that historically caused failures and add tests that would have caught them. |
中文
测试是执行软件以检查其行为是否正确的活动——这里的“正确”，指的是符合某种规约，无论这种规约是形式化还是非形式化。这个定义包含一个重要限制：测试只能证明 bug 的存在，不能证明 bug 的不存在。Edsger Dijkstra 在 1972 年说过这一点，其逻辑很直接。一个程序对所有输入的正确性需要证明；一个测试只能证明它在被测试的特定输入上正确。除非输入空间有限且足够小，可以被完全穷尽，否则测试就是抽样，而抽样天然存在覆盖缺口。问题不是要不要测试——测试是必要的——而是如何把测试做得足够好，使覆盖缺口可管理，而不是灾难性的。
软件测试这门纪律为这个问题提供结构化回答：应该写哪些测试类型，以什么比例写，在什么抽象层级写，用什么工具写，使用什么输入生成策略，以及如何把测试整合进开发流程，使它提供持续反馈，而不是一次性关卡。过去二十年里，这些答案发生了很大变化，推动因素包括自动化测试基础设施的发展、测试驱动开发作为实践的出现、持续集成作为部署模型的普及，以及基于属性的测试作为生成开发者想不到的输入的方法。
Quality assurance 是更宽泛的纪律，testing 是其中一部分。QA 包括定义 quality attributes——correctness、reliability、performance、security、maintainability、usability——并建立产生这些属性的软件实践和流程。Testing 提供关于 correctness 以及已知 defect types 是否存在的证据；QA 则提供框架，用来询问这些证据是否充分，还有哪些 quality attributes 需要注意，以及开发流程应如何被组织，以预防 defects，而不是只检测 defects。
前置知识：编程（§2.1）——写 tests 也是编程。算法（§2.6）——理解 computational complexity 有助于推理 test coverage。软件架构（§7.1）——testability 是必须被设计进去的架构属性。
从 Ad Hoc Verification 到 Test-Driven Development
自从有软件可测以来，软件就一直被测试；但把 testing 作为一门纪律进行系统研究，是从 1970 年代后期开始的。
最早的软件测试是探索式的：写程序的程序员会用一些 inputs 运行程序，并检查 output 看起来是否正确。这并不是疏忽——这是当时唯一可用的选项。系统性测试工具还不存在，test coverage 理论尚未发展出来，automated regression testing 的概念也还没有被清楚提出。1950 年代和 1960 年代产生了第一代程序员，他们知道 debugging 很昂贵，也知道在开发后期发现 bugs 比早期发现更昂贵得多，但他们还没有系统框架来做这件事。
Gerald Meyers 的 The Art of Software Testing（1979）是第一部把软件测试作为工程纪律系统处理的著作。Meyers 阐明了 black-box testing（通过组件外部接口测试组件，不了解其实现）与 white-box testing（在了解实现的情况下测试，以覆盖特定 code paths）之间的概念区别。他提出 equivalence partitioning——把输入空间划分为应产生相同结果的 input classes，然后从每个 class 中测试一个代表值——作为系统性选择测试输入的方法。他还阐明了 boundary value analysis——在 input ranges 的边界处测试，因为 bugs 最常出现在那里——作为互补技术。这些原本为手工 test case design 发展出的概念，半个世纪后仍然是系统性测试设计的基础。
Code coverage 的概念——测量 test suite 执行了代码中多少 branches、statements 或 paths——在 1970 年代作为 testing adequacy 的指标出现。Branch coverage（也叫 decision coverage）衡量每个 conditional 的 true 和 false branches 是否至少被一个 test 执行过。这为测试彻底性提供了可计算 proxy，尽管这个 proxy 并不完美。一个拥有 100% branch coverage 的 test suite 仍然可能漏掉 bugs——coverage 衡量的是哪些 code 被执行了，而不是被执行的 code 是否被测试了正确行为。
1970 年代和 1980 年代的软件危机——大型软件项目持续延期、超预算，并带有大量 defects——推动软件工程纪律更严肃地对待 testing。V-model of development 把每个开发阶段与一个相应 testing phase 配对（unit testing 对应 detailed design，integration testing 对应 high-level design，system testing 对应 requirements），从而形式化了测试与开发阶段之间的关系。这个模型被政府和航空航天标准采用，并塑造了大型组织数十年来组织 testing 的方式。
1980 年代后期和 1990 年代 object-oriented programming 的出现，带来了新的 testing 挑战。Object-oriented code 比 procedural code 更难 isolation testing，因为 objects 有 state，而 state 会通过一系列 method calls 累积。Setup（在 test 前建立 state）、exercise（执行被测行为）和 verify（检查结果是否正确）这些概念，成为标准 test structure。Kent Beck 在 1990 年代早期从事 Smalltalk 工作时，开发了 SUnit，这是第一个实现这种结构的 unit testing framework；JUnit 则由 Beck 和 Erich Gamma 于 1990 年代中期为 Java 开发，把 xUnit pattern 带给了更广泛受众。JUnit 普及了把所有 tests 作为 build 一部分自动运行的概念，使 automated regression testing 变得实用。
Martin Fowler 和 Kent Beck 对 continuous integration 的阐述——也就是每天多次把 code changes 集成到 shared repository 中，并通过包含 tests 的 automated build 进行验证——形式化了 automated testing 与软件开发实践之间的联系。Fowler 2000 年的文章 “Continuous Integration”（后来在 2006 年扩展）描述了这项实践及其收益：因为 integration 频繁发生，integration problems 会立即被发现，而不是在数周并行开发中不断积累。每次 integration 时运行的 automated test suite，提供了 integration 没有破坏既有行为的信号。Continuous integration 使从 code 到 test result 的 feedback loop 足够快，testing 因而成为开发中的自然部分，而不是开发结束后的一个阶段。
Test-Driven Development（TDD）由 Kent Beck 在 Test-Driven Development: By Example（2002）中形式化，它颠倒了写 code 与写 tests 的常规关系。在 TDD 中，developer 会在写实现代码之前，先为期望行为写一个 test，看着这个 test fail（确认 test 确实在测试某些东西），然后写最少代码让 test pass，最后在保持 tests passing 的情况下 refactor code。这个循环——Red、Green、Refactor——产生的是从诞生起就被测试的代码，其 testability 被内置其中（因为它本来就是为了被测试而写），而且实现倾向于最小化，因为只写足够通过当前 tests 的代码。TDD 在降低 defect rates 上有经验支持，并影响了一代 developers 接近设计的方式；它也被批评在预期行为无法事先知道的领域中不现实，并且容易鼓励过度依赖 unit tests，而牺牲 integration 和 end-to-end tests。
Property-based testing 由 Koen Claessen 和 John Hughes 在 2000 年论文 “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs” 中提出，它为“应该测试哪些 inputs”这个问题提供了不同答案。Developer 不是指定单个 examples，而是指定 properties——关于 code 应该做什么的一般性陈述——testing framework 则生成 random inputs 来寻找 counterexamples。Haskell 的 QuickCheck 催生了多数主流语言中的实现（Python 的 Hypothesis、JavaScript 的 fast-check、Java 的 jqwik）。Property-based testing 能发现 example-based tests 漏掉的 bug 类别，因为它会生成 developer 没有想到的 inputs；其限制在于，指定有用 properties 需要比指定 examples 更抽象的思考。
Testing pyramid 由 Mike Cohn 在 Succeeding with Agile（2009）中描述，并由 Fowler 等人进一步阐发。它提出 test suites 应包含大量 unit tests、较少 integration tests，以及更少的 end-to-end tests。金字塔形状反映的是速度与成本权衡：unit tests 以毫秒运行，并能精确隔离失败；end-to-end tests 以秒或分钟运行，而且失败后需要大量调查才能识别原因。作为处方，pyramid 后来被不断修正——“testing trophy”（由 Kent C. Dodds 普及）更强调 integration tests——但底层原则仍然可靠：不同 test types 服务不同目的，其比例应根据这些目的的成本和收益来校准。
当前时期的 test infrastructure，在过去几代人看来会很惊人：continuous integration systems 会在每个 commit 上运行完整 test suite；coverage measurement 会追踪随时间变化的趋势；mutation testing 会通过注入 deliberate bugs 并检查 tests 是否抓住它们来评估 test quality；contract testing 会验证 services 之间的 assumptions；observability infrastructure 会使生产行为可测量。用于 test generation 的 AI assistance——建议 test cases、编写 test scaffolding、生成 property specifications——正在被整合进开发工具。基本挑战并没有改变：input space 是无限的，oracle problem（知道正确 output 应该是什么）很难，而写出 maintainable、fast、reliable 的 tests 仍然是一门需要技能和判断的纪律。
Test Types、Coverage 与 Test Quality
Test Hierarchy 及每个层级提供什么
Unit tests 会在与依赖隔离的情况下验证单个 functions 或 classes 的行为。一个计算第 n 个 Fibonacci number 的 function，一个验证 email addresses 的 class，一个把 string 转换为 date 的 parser——每一个都可以被独立测试，inputs 由程序提供，outputs 则与 expected values 比较。Unit tests 很快（毫秒级）、deterministic（相同 inputs 产生相同 outputs），并且隔离性强（失败 test 指向特定 function 或 class）。其限制是，它们不测试 components 之间的互动；一个所有 unit tests 都 pass 的系统，在 units 被组合起来时仍然可能失败。
Test doubles 是总称，包含 stubs（返回预设 values）、mocks（验证某些 calls 是否发生）、spies（记录 calls）、fakes（实现依赖的简化版本）和 dummies（填充 parameter lists，但从不使用）。它们允许 unit tests 把被测 component 与其 dependencies 隔离开来。一个发送 email 的 function，可以通过提供一个 mock email sender 来测试，该 mock 只记录 calls，而不会真正发送 email。风险在于，如果 test doubles 与它们替代的真实 components 同步不好，那么 tests 会 pass，但真实系统会失败；components 之间的 contract tests 可以缓解这一点。
Integration tests 验证 components 一起工作时的行为，通常使用真实 dependencies，而不是 mocked dependencies。一个测试 REST endpoint、向正在运行的 service 发送真实 HTTP request，并检查 HTTP response 的 test，就是 integration test。Integration tests 比 unit tests 慢（它们可能涉及 network calls、database queries、file operations），也更难隔离（一次失败可能涉及任何参与互动的 components），但它们能验证 unit tests 不能验证的行为：components 是否正确解释彼此的数据格式，是否处理彼此的错误条件，是否同意 shared interfaces 的语义。
End-to-end tests 通过用户界面验证完整系统行为。一个 Playwright 或 Selenium test 登录、导航到表单、填写并提交，然后检查 resulting state，就是 end-to-end test。End-to-end tests 对 user-visible behavior 提供最真实的验证，但也是最昂贵且最 flaky 的：它们依赖 browser behavior、timing、network conditions，以及 UI 调用的所有 services。一个覆盖最关键 user journeys 的小型 end-to-end tests suite 有价值；大型 suite 则会变成维护负担，而且失败难以诊断。
Contract testing，尤其是 Pact 实现的 consumer-driven contract testing，会验证一个 service 对另一个 service API 的 assumptions。Consumer 定义自己预期的 interactions（request format、response format、status codes）；provider 验证自己能满足这些 interactions。这使团队可以在不同时运行所有 services 的情况下测试 service boundaries，并在 API incompatibilities 进入生产前发现它们。
Mutation testing 通过向 source code 中引入 deliberate bugs（mutations）来评估 test quality——把 + 改成 -，把 > 改成 >=，把 true 改成 false——然后检查 test suite 是否抓住 mutation（mutation 被 “killed”）还是漏掉它（mutation “survives”）。Surviving mutations 表示 test suite 没有充分验证 mutated code 所实现的行为。Mutation testing 计算成本高，但它提供了比 code coverage 更有意义的 quality metric：coverage 衡量 execution，mutation testing 衡量 verification。
Test Coverage 及其限制
Coverage metrics——statement、branch、path、mutation——衡量 test suite 以不同方式覆盖代码的程度。Branch coverage（每个 conditional 的 true 和 false branch 是否都被执行过？）是最常要求的 metric，也是简单 metrics 中最有意义的。100% branch coverage 对多数代码来说可达，也可以作为 critical code 的合理最低标准；但它并不保证没有 bugs。
Coverage 是 test quality 的下限，不是上限。一个执行每个 branch、但不 assert meaningful outputs（或只 assert 没有抛出 exception）的 test，拥有完整 coverage，却什么也没有测试。Coverage tools 可以被游戏化：一个想达到 coverage target 的 developer，可以写出执行代码但不 assert 任何东西的 tests。有意义的标准不是“哪些 code 被执行了”，而是“哪些 behaviors 被验证了”。
Oracle problem——正确 output 应该是什么的问题——才是 test quality 的最终所在。对于许多 functions，正确 output 很容易指定：两个 integers 的和、一个 list 的排序结果。对另一些情况，它需要难以编码的 domain knowledge：一张分类图像是否正确，一段生成文本是否可接受，一条 recommendation 是否相关。Property-based testing 通过指定 invariant properties（output 是 input 的 permutation，result 是 non-negative），而不是 exact outputs，部分缓解了 oracle limitations；但即使是 properties，也需要谨慎思考才能指定。
Test Design Principles
难以维护的 tests 最终会被放弃。Test maintainability 的主要敌人，是与 implementation 耦合而不是与 behavior 耦合（implementation 改变但 behavior 不变时仍然失败的 tests）、test structure 不清楚（从 code 中看不出 test 目的），以及 isolation 不足（tests 的 pass 或 fail 取决于执行顺序或 external state）。
Arrange-Act-Assert（AAA）pattern 会使 tests 结构清楚：arrange preconditions 和 inputs；act，执行被测 behavior；assert，检查 postconditions 是否正确。每个 test 理想上应测试一个 behavior；一个 test 失败时，应指向一个具体问题，而不是几个可能问题中的任何一个。Test names 应描述被验证的 behavior（“when_user_email_is_invalid_registration_should_fail”），而不是被执行的 mechanics（“test_registration_with_invalid_email”）。
依赖 shared mutable state 的 tests 是 flaky 的：它们会根据之前运行了哪些 tests，以及以什么顺序运行而 pass 或 fail。每个 test 都应建立自己需要的 state，并清理自己创建的任何 state。Database tests 尤其容易出现这个问题；让每个 test 运行在一个结束时会 rollback 的 transaction 中，或在每次 test run 前 truncate 并 seed database，都可以提供 isolation。
学习这一部分会改变什么
测试会改变实践者在开发过程中，而不是开发之后，思考代码的方式。
第一个变化，是把 testability 当成 design constraint。难以测试的 code 也难以修改：使 function 难以 isolation testing 的同一 coupling，也使它难以在不理解和修改所有 dependencies 的情况下被改变。在设计阶段思考 testing 的实践者，会写出带 injected dependencies（而不是 hard-coded dependencies）的代码、小而输入输出清楚的 functions，以及 components 之间定义良好的 boundaries。结果是代码既更 testable，也更 maintainable。
第二个变化，是能够区分 test types，并恰当使用它们。一个只知道“tests 放在这里”的 developer，会在最方便的层级写 tests，而不是在最合适的层级写。理解 unit、integration 和 end-to-end tests 不同目的的实践者，可以构建一个 test suite，使每种 test type 都提供特定价值，并且其比例根据成本和收益校准。
第三个变化，是认真对待 failures 的纪律。一个被部分忽视的 test suite——“我们有 500 个 failing tests，就是这样”——没有价值，而且具有误导性：它暗示系统经过测试，而实际并没有。认真对待 testing 的实践者会保持 test suite green：每个 failing test 要么被修复，要么被明确标记为 known-failing，并关联 tracked issue。维持这种纪律比听起来更难，而它正是 testing culture 与 testing checkbox 之间的区别。
第四个变化，是以概率方式思考 tests 可能发现哪些 bugs。Example-based tests 会发现 developer 想到的 bugs；property-based tests 会发现 examples 之间空间里的 bugs；mutation testing 会揭示 tests 的 assertions 在哪里比 coverage 所暗示的更弱。理解 testing strategy 与每种策略能发现、不能发现的 bug 类型之间关系的实践者，更能根据系统 risk profile 设计合适 test suite。
资源
书籍与文本
Meyers、Sandler 和 Badgett 的 The Art of Software Testing（第 3 版，Wiley，2012）是基础参考，从 1979 年原版更新而来。它对 test case design 的系统方法——equivalence partitioning、boundary value analysis、cause-effect graphing、error guessing——仍然适用于任何测试工作。它发展的 black-box techniques 独立于 implementation language 或 framework。
Beck 的 Test-Driven Development: By Example（Addison-Wesley，2002）是 TDD 奠基文本。书中的两个 worked examples（Java 中的 money calculation，以及 Python 中的 JUnit implementation）展示了 TDD cycle 在实践中如何运作。该书对 testing culture 影响巨大；阅读它可以直接接触这项实践背后的推理。
Khorikov 的 Unit Testing: Principles, Practices, and Patterns（Manning，2020）是当代关于良好 unit testing 原则最有用的文本。它的区分——观察 behavior 与 implementation details 的区别、high-value tests 与 low-value tests 的区别、sociable 和 solitary unit tests 的区别——提供了一个超越 coverage metrics 的 test quality 评估框架。关于 classical versus London schools of TDD（在哪里使用 mocks，在哪里不使用）的讨论，是对这一争议最清楚的处理。
Freeman 和 Pryce 的 Growing Object-Oriented Software, Guided by Tests（Addison-Wesley，2009）展示了 integration level 的 TDD，从一个 end-to-end acceptance test 开始，并通过 test-first discipline 推动 components 设计。它的 outside-in approach 与 Beck 的 inside-out approach 形成对比，处理了从带 mocks 的 unit tests 开始的局限。
Humble 和 Farley 的 Continuous Delivery（Addison-Wesley，2010）把 testing 放入 deployment pipeline 中，覆盖 automated test stages（unit、integration、acceptance、performance）、每个 stage 的运行条件，以及设计一个能对每个 commit 提供快速反馈的 pipeline。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Meyers, Sandler & Badgett, The Art of Software Testing（第 3 版） | 基础 test design techniques | 深入 |
| Beck, Test-Driven Development: By Example | TDD 奠基文本 | 入门 |
| Khorikov, Unit Testing: Principles, Practices, and Patterns | 当代 unit testing 原则 | 实践 |
| Freeman & Pryce, Growing Object-Oriented Software, Guided by Tests | TDD outside-in approach | 深入 |
| Continuous Delivery testing chapters（§7.4 参考） | Deployment pipeline 中的 testing | 深入 |
| Osherove, The Art of Unit Testing（第 3 版） | 实践性 unit testing 参考 | 入门 |
课程、论文与当前资料
Dijkstra 的 “Notes on Structured Programming”（1972，免费在线）包含那句常被引用的话：“program testing can be used to show the presence of bugs, but never to show their absence.” 阅读其上下文——Dijkstra 关于程序数学验证的更广泛论证——可以帮助理解 testing 能主张什么、不能主张什么。
Fowler 在 martinfowler.com 上的 testing articles（免费）——尤其是 “UnitTest”、“IntegrationTest”、“TestDouble”、“TestingStrategy” 和 “Mocks Aren’t Stubs”——是当代对 testing concepts 及其区别最谨慎的处理。Ham Vocke 在 martinfowler.com 上的 “Practical Test Pyramid” 文章，则把 pyramid 应用于当代 Web application testing。
Claessen 和 Hughes 的 “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs”（2000，可在 ACM DL 免费获取）是引入 property-based testing 的论文。阅读它可以获得这项技术独立于具体实现的概念基础。
| 资源 | 平台 | 类型 |
|---|---|---|
| Dijkstra, “Notes on Structured Programming”（1972，免费） | Testing 限制的基础文本 | 深入 |
| Fowler, testing articles（免费） | martinfowler.com | 参考 |
| Claessen & Hughes, QuickCheck paper（免费） | Property-based testing 基础 | 深入 |
| Google Testing Blog / Tech on the Toilet（免费） | testing.googleblog.com | 辅助 |
实践、工具与当前资料
每种主流 programming language 都有标准 testing frameworks。JUnit 5（Java）、pytest（Python）、Jest（JavaScript/TypeScript）、RSpec（Ruby）、Go testing package——具体 framework 重要性低于在你的主力语言所用 framework 中发展熟练度，包括 parameterized tests、fixtures，以及 test runner output 的理解。
Coverage tools——JaCoCo（Java）、Coverage.py（Python）、Istanbul/nyc（JavaScript）——测量哪些 branches 和 statements 被执行。把 coverage measurement 整合进 CI，并追踪 coverage trends over time，会使 coverage 成为有意义信号，而不是一次性评估。
Hypothesis（Python，免费）和 fast-check（JavaScript，免费）是成熟 property-based testing libraries。为 sorting function、parser 或 data validation library 这类 properties 清楚的对象编写 property-based tests，可以直接体验 property-based testing 如何发现 example-based tests 漏掉的 bugs。
Pact（免费，pact.io）在多种语言中实现 consumer-driven contract testing。完成 Pact example——定义 consumer contract，并用它验证 provider——可以使 contract testing workflow 变得具体。
PIT（Java mutation testing，免费）和 Stryker（JavaScript/TypeScript mutation testing，免费）可以在既有 test suite 上运行 mutation testing。对自己写过的 codebase 运行 mutation testing，并检查 surviving mutants，是发现 test suite assertions 哪里弱于 coverage 所暗示内容的最快方式。
| 资源 | 平台 | 类型 |
|---|---|---|
| JUnit 5 / pytest / Jest（免费） | Language-specific | 实践 |
| Coverage tools: JaCoCo / Coverage.py / Istanbul（免费） | Language-specific | 实践 |
| Hypothesis（Python，免费） | hypothesis.works | 实践 |
| fast-check（JavaScript，免费） | GitHub | 实践 |
| Playwright Test（免费） | playwright.dev | 实践 |
| Pact contract testing（免费） | pact.io | 实践 |
| PIT mutation testing（Java，免费） | pitest.org | 实践 |
| Stryker（JS/TS mutation testing，免费） | stryker-mutator.io | 实践 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 把 coverage 当成 quality proxy | 即使 tests 什么也不 assert，100% branch coverage 也可以达到。一个 test suite 可以 coverage 很高但 quality 很低：它执行了代码，却没有验证代码做了正确的事。Coverage 是 test suite quality 的必要但不充分条件。 | 通过 mutation testing（tests 是否能抓住 deliberate bugs？）和 test review（每个 test 是在验证 behavior，还是只是在执行 code？）评估 test quality。把 coverage thresholds 设为下限，而不是上限，并把 mutation score 与 coverage 一起用于评估 suite quality。 |
| Unit level 的 over-mocking | 把 unit under test 的每个 dependency 都 mock 掉的 tests，验证的是这个 unit 是否发出了它会发出的 calls，而不是这些 calls 是否产生正确结果。当真实 dependencies 被 mocks 完全替代时，integration problems 对 test suite 不可见。“London school” approach——用 mocks 把 units 完全隔离测试——会产生 brittle 的 test suites（任何 dependency 变化都需要更新 mocks），并漏掉 integration bugs。 | 在测试 components 之间互动时，优先使用 integration tests。有选择地使用 mocks，只用于那些 slow、non-deterministic，或具有 external side effects 的 dependencies（发送 email、payment processing）。Fowler 和 Khorikov 关于何时 mock、何时使用真实 dependencies 的处理，提供了做这些决策的框架。 |
| 测试 implementation 而不是 behavior | 一个 test 如果验证某个 function 会执行特定内部 method calls 序列，那么即使 visible behavior 没有改变，只要 implementation 被 refactor，它也会失败。这类 tests 是 refactoring 的税：每次内部变化都要求更新 tests，无论 behavior 是否变化。 | 针对 public interface 写 tests，而不是针对 implementation。对 sorting function 的 test 应验证 output 已排序，且是 input 的 permutation；它不应验证 function 会在调用 quicksort 前先调用 partition。只要通过 public interface 可观察到的 behavior 被保留，implementation 就应能自由变化。 |
| Test orphan problem | 写出之后不维护的 tests 会越来越误导：它们可能在 code 已变化后仍然 pass，或因与被测 behavior 无关的原因 fail。一个大多被忽视的 failing test suite（因为“那些 tests 总是失败”）比没有 test suite 更糟：它提供了“我们有 tests”的组织安慰，却没有真正 safety net。 | 对 test failures 采取零容忍政策：suite 中每个 test 要么 pass，要么被明确标记为 known failure 并关联 tracked issue。如果一个 test 持续因不是 bugs 的原因失败，就删除它或修复它。小而 green 的 test suite 比大而 red 的 test suite 更有价值。 |
| 只测试 happy path | 只覆盖预期、well-formed inputs 和 successful execution paths 的 tests，会漏掉导致生产失败的 bugs：null inputs、empty lists、unexpected encodings 中的 strings、concurrent access、network failures、database constraint violations。生产系统会遇到完整输入范围，而不只是 developer 心中干净的输入。 | 系统性应用 boundary value analysis 和 equivalence partitioning：empty inputs 会怎样？valid ranges 边界处的 inputs 会怎样？组合多个 edge conditions 的 inputs 会怎样？使用 property-based testing 生成 developer 想不到的 inputs。回顾 bug reports 中历史上导致 failures 的输入类型，并添加本可抓住它们的 tests。 |