English
Software testing checks a system’s behavior by executing it under selected conditions and comparing the observations with expectations. The expectations may concern outputs, state changes, errors, timing, or interactions with another system. A test needs both a way to exercise behavior and an oracle: a basis for deciding whether what happened was acceptable.
Suppose delivery is free for orders of at least 100 currency units. A test can submit an order worth 100 and check that its delivery charge is zero. This checks one requirement at one boundary. It does not establish that cancelled orders are refunded correctly or that two simultaneous updates preserve the total. Good testing starts by identifying the behavior and failure being investigated.
Quality assurance is broader than test execution. Requirements review, code review, static analysis, build controls, and learning from production failures can prevent or reveal problems that a particular test suite misses. Verification asks whether the implementation meets specified requirements; validation asks whether those requirements and the resulting system serve the intended use. A perfectly implemented rule can still be the wrong rule for users.
Prerequisites: Programming (§2.1) and basic control flow. Architecture (§7.1) helps identify boundaries; databases and concurrency become relevant when the behavior under test depends on them. Begin with a small deterministic function before testing a complete service.
From Ad Hoc Verification to Test-Driven Development
Early programmers already used examples, expected results, and checking routines. As programs grew, tests had to become repeatable and survive changes to the code. A regression test records behavior that should continue to work, so that fixing one problem is less likely to reintroduce another unnoticed.
Dijkstra’s discussion of structured programming emphasized the difference between observing successful executions and establishing a general claim about a program. Ordinary testing samples an input and execution space that is often too large to exhaust. Exhaustive testing is possible for some finite, precisely bounded models, but its conclusion still concerns that model and its assumptions. This distinction connects testing to the specifications and formal methods of §3.6.
Glenford Myers’s The Art of Software Testing (1979) organized techniques such as equivalence partitioning and boundary-value analysis. Inputs are grouped according to behavior relevant to a requirement, then representatives and boundaries are tested. This is a design hypothesis about where defects may hide, not proof that every input in a group behaves identically. Coverage tools later made it easier to see which parts of an implementation a test suite had exercised.
Stateful objects and reusable libraries increased the need to establish test state and check sequences of operations. Beck’s SUnit for Smalltalk and Beck and Gamma’s JUnit for Java helped popularize the xUnit structure of repeatable setup, execution, and checks. They were influential testing frameworks, not the beginning of automated testing. Continuous integration made these checks useful during ongoing development by testing combined changes frequently.
Test-driven development uses a short cycle: specify a behavior with a failing test, implement enough to satisfy it, then improve the structure while preserving behavior. Observing the initial failure helps check that the test can detect the missing behavior; a failure caused only by a broken fixture provides different evidence. TDD can support design and regression testing, but does not independently establish that the requirements or test oracle are correct.
Claessen and Hughes’s 2000 QuickCheck paper popularized executable properties combined with generated inputs. Modern property-testing tools can also reduce a failing input to a smaller counterexample. Contract tests address assumptions between separately developed components, while mutation testing asks whether tests detect selected changes to the implementation. These techniques answer different questions; none makes the choice of meaningful expectations unnecessary.
Test Types, Coverage, and Test Quality
The Test Hierarchy and What Each Level Provides
A unit test focuses on a small unit of behavior, which may be a function, class, or a few collaborating objects. Isolation can mean independence from other tests and uncontrolled resources; it need not mean replacing every collaborator. A local pricing calculation is a good candidate. Fast execution is useful, but neither speed nor determinism follows merely from calling something a unit test.
Test doubles replace dependencies for a particular purpose. A stub supplies chosen responses, a spy records calls, a mock checks expected interactions, and a fake supplies a simplified working implementation. A controlled clock lets a test cross an expiry boundary without waiting. A fake email sender avoids real messages. Doubles can disagree with the production dependency, so they should not be the only evidence that an integration works.
Integration tests exercise a boundary with the relevant components together. A database test can check whether two concurrent reservation attempts really allow only one commit. An in-memory fake that has no transactions cannot establish that property. End-to-end tests exercise a complete journey through the system’s external interface, which may be a browser, API, or command line. They check deployed components and configuration together, at greater setup and diagnostic cost.
Consumer-driven contract tests record interactions a consumer expects and verify them against a provider. They can detect incompatible request or response changes without running the entire application. Their scope is the recorded contract and provider states; they do not automatically check network reachability, production credentials, throughput, or every possible sequence of interactions.
Test pyramids and other suggested portfolios are cost models, not fixed quotas. Use focused tests for many combinations of a local rule, integration tests for real boundary semantics, and complete journeys for critical workflows. Performance, security, and accessibility are concerns that can require tests at several of these levels. The right balance depends on which failures matter and where they can be detected reliably.
Test Coverage and Its Limitations
Statement coverage records which statements ran; branch coverage records which control-flow alternatives ran. Coverage reports are useful for finding unexercised code, but do not measure the correctness of assertions or the completeness of requirements. Some branches are infeasible under valid inputs, and path counts can grow rapidly with conditions and loops. A universal percentage target obscures these differences.
For the free-delivery rule, suppose the implementation mistakenly uses total > 100 instead of total >= 100. Tests at 99 and 101 exercise both outcomes of the condition and can assert correct charges, yet miss the defect at 100. Even full branch coverage can miss an important boundary. This example is stronger than a test that merely executes code without checking anything: meaningful assertions can still be incomplete.
Mutation testing deliberately changes an operator or expression and reruns the tests. A surviving change deserves investigation, but may be equivalent to the original over the valid input domain, or may affect irrelevant behavior. A mutation score therefore depends on which mutations are generated, which are excluded, and how equivalent cases are handled. It is another diagnostic, not a percentage of real bugs detected.
When an exact oracle is hard to obtain, use independent constraints. For sorting, check both nondecreasing order and preservation of each input element’s multiplicity. An implementation that always returns an empty list satisfies the first property and fails the second on nonempty inputs. A parser and serializer passing a round-trip test can still share the same misunderstanding of an external format; examples from that format’s specification provide an independent check.
Test Design Principles
Describe the requirement before choosing test data. For the threshold rule, use 99, 100, and 101; then decide whether zero, negative amounts, fractions, and missing values belong to the valid domain or should be rejected. Equivalence classes and boundaries should come from the contract as well as the implementation. Property-based testing adds generated cases and shrinking, but depends on the generators reaching relevant situations and on the property being useful.
Arrange the required state, perform the operation, and assert its observable consequences. One behavior may require several assertions: a cancelled reservation may release the seat and create exactly one refund request. Avoid asserting internal call sequences unless that sequence is part of a real contract. Give tests enough context that a failure explains what requirement may have been violated.
Control clocks, random seeds, test data, and asynchronous completion where appropriate. Use explicit synchronization instead of assuming a task finishes after an arbitrary sleep. A transaction rolled back after a test isolates only work that participates in that transaction; another process or connection may commit independently. Parallel tests also need separate resources or deliberate coordination. A flaky result may reveal a real race, so investigate before classifying it as test noise.
Reliability matters at suite scale. If 100 tests each independently have a 0.1% chance of a spurious failure, the chance of at least one such failure in a run is 1 − 0.999¹⁰⁰ ≈ 9.52%. Real failures need not be independent, but the calculation explains why individually rare problems can make a suite hard to trust. Rerunning until green hides this signal. Temporary quarantine should have an owner and a repair plan while preserving the failing evidence.
AI-generated tests require the same review. Tests copied from an implementation can repeat its mistakes, including the wrong threshold. Supply requirements and counterexamples independently, inspect the assertions, and deliberately run a faulty version when practical to see whether the test detects the intended error.
What Studying This Changes
You learn to ask what evidence a passing test actually provides. A test of a pricing rule, a test against a real database, and a complete checkout journey support different claims. This makes it easier to add a missing check at the appropriate boundary instead of increasing test count without a purpose.
Start with the delivery-charge example: obtain full branch coverage with 99 and 101, introduce the incorrect comparison, and observe that those tests still pass. Add 100 and see the defect become visible. Then test a deliberately broken sorter with an ordering property alone and with a multiplicity property added. These exercises separate execution coverage from the strength of a test oracle.
For a small application, choose one important failure path, such as cancellation during a retry, and state the required result before writing tests. After a real bug, add a focused regression test and consider whether its cause suggests a broader property or a missing integration test. Keep the suite interpretable: a failure should lead to investigation, not routine dismissal.
Resources
Books and Texts
Myers, Sandler, and Badgett develop systematic test-case design in The Art of Software Testing. Beck demonstrates the TDD cycle through examples; Freeman and Pryce develop an outside-in approach. Khorikov discusses test value and coupling to implementation. Osherove’s The Art of Unit Testing, third edition, is written with Vladimir Khorikov and uses JavaScript examples. Compare the assumptions of these approaches rather than treating one testing style as mandatory.
| Book | Role | Type |
|---|---|---|
| Myers, Sandler & Badgett, The Art of Software Testing (3rd ed.) | Systematic test-case design | Depth |
| Beck, Test-Driven Development: By Example | TDD foundational text | Entry |
| Khorikov, Unit Testing: Principles, Practices, and Patterns | Contemporary unit testing principles | Practice |
| Freeman & Pryce, Growing Object-Oriented Software, Guided by Tests | TDD outside-in approach | Depth |
| Continuous Delivery testing chapters (§7.4 reference) | Testing in deployment pipeline | Depth |
| Osherove with Khorikov, The Art of Unit Testing (3rd ed.) | Unit testing with JavaScript examples | Entry |
Courses, Papers, and Current Sources
Read Dijkstra’s structured-programming discussion for the distinction between tests and general correctness arguments. The QuickCheck paper explains properties and generated examples; Hypothesis documentation makes generation and counterexample reduction practical. Pact documentation shows how consumer expectations are checked against providers.
| Resource | Platform | Type |
|---|---|---|
| Dijkstra, Notes on Structured Programming (1969 manuscript; published 1972) | Program reasoning and the scope of testing | Depth |
| Fowler, testing articles (free) | martinfowler.com | Reference |
| Claessen & Hughes, QuickCheck paper (free) | Property-based testing foundation | Depth |
| Google Testing Blog / Tech on the Toilet (free) | testing.googleblog.com | Auxiliary |
Practice, Tools, and Current Sources
Use the test runner already suited to your project, then add parameterized boundary cases and one useful property. Examine a coverage report and deliberately introduce a defect that coverage alone misses. Mutation tools can automate some such changes. For integration and browser tests, keep data and external effects under control and retain enough diagnostic information to explain a failure.
| Resource | Platform | Type |
|---|---|---|
| JUnit / pytest / Jest documentation | Language-specific test frameworks | Practice |
| Coverage tools: JaCoCo / Coverage.py / Istanbul (free) | Language-specific | Practice |
| Hypothesis (Python, free) | hypothesis.works | Practice |
| fast-check (JavaScript, free) | GitHub | Practice |
| Playwright Test (free) | playwright.dev | Practice |
| Pact contract testing (free) | pact.io | Practice |
| PIT mutation testing (Java, free) | pitest.org | Practice |
| Stryker (JS/TS mutation testing, free) | stryker-mutator.io | Practice |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Equating coverage with correctness | Executed branches can have incomplete assertions or missing boundary cases. | Inspect requirements and counterexamples alongside coverage. |
| Replacing every dependency with a mock | The doubles may reproduce assumptions that the real integration violates. | Use doubles deliberately and test important boundaries with real components. |
| Testing incidental implementation details | Internal rearrangements cause failures even when required behavior is unchanged. | Assert outputs, state transitions, and interactions that belong to the contract. |
| Rerunning unexplained failures until green | Retries can conceal nondeterminism and actual races. | Keep evidence, investigate, and assign ownership to temporarily quarantined tests. |
| Testing only successful examples | Invalid input, interruption, concurrency, and recovery can have different behavior. | Select cases from the contract, important boundaries, and plausible failure modes. |
中文
软件测试通过在选定条件下执行系统，把观察到的行为与预期比较。预期可以涉及输出、状态变化、错误、时间或与其他系统的交互。测试既需要触发行为的方法，也需要测试判据：凭什么判断发生的事情是否可以接受。
假设订单金额达到 100 就免运费。测试可以提交金额为 100 的订单，检查运费为零。这检查了一个需求的一个边界，却不能证明取消订单时退款正确，也不能证明两个并发更新保持总额正确。好的测试从明确要检查的行为和失败方式开始。
质量保证的范围比执行测试更广。需求审查、代码审查、静态分析、构建控制和对线上故障的分析，都可能预防或发现某个测试集遗漏的问题。验证关注实现是否符合规定的要求；确认关注这些要求及系统是否满足实际用途。即使规则被完全正确地实现，它仍可能不适合用户。
前置知识：编程（§2.1）与基本控制流。架构（§7.1）帮助确定边界；被测行为涉及数据库或并发时，再引入相应知识。可以先测试小型确定性函数，再测试完整服务。
从 Ad Hoc Verification 到 Test-Driven Development
早期程序员已经使用输入示例、预期结果和检查例程。程序扩大后，测试需要能够重复运行，并在代码修改后继续使用。回归测试记录应该持续成立的行为，降低修复一个问题时悄悄重新引入另一个问题的可能性。
Dijkstra 关于结构化程序设计的讨论，强调了观察到成功执行与证明程序的一般性质之间的区别。常规测试只抽取输入和执行空间的一部分，而这个空间通常大得无法穷尽。对于某些有限且边界明确的模型，可以进行穷尽测试，但结论仍然只涉及该模型及其假设。这一区别把测试与 §3.6 的规范和形式化方法联系起来。
Glenford Myers 在 1979 年的《The Art of Software Testing》中整理了等价类划分和边界值分析等技术。测试者按与需求有关的行为对输入分组，再检查代表值与边界。这是关于缺陷可能藏在哪里的设计假设，并不能证明同组输入的行为完全一样。覆盖率工具随后使测试集执行过哪些实现部分更容易观察。
有状态对象和可复用库增加了建立测试状态、检查操作序列的需要。Beck 为 Smalltalk 编写的 SUnit，以及 Beck 和 Gamma 为 Java 编写的 JUnit，推广了可重复的准备、执行与检查结构。它们是有影响力的测试框架，而不是自动化测试的起点。持续集成通过频繁检查合并后的修改，让这些测试参与日常开发。
测试驱动开发使用短循环：先用失败的测试描述行为，再实现足以通过测试的代码，随后在保持行为的前提下改进结构。观察最初的失败，有助于确认测试能发现缺失的行为；如果只是测试准备代码出错，提供的证据就不同。TDD 可以支持设计和回归测试，却不能单独证明需求或测试判据正确。
Claessen 和 Hughes 在 2000 年发表的 QuickCheck 论文，推广了把可执行性质与生成输入结合的方法。现代性质测试工具还可以把失败输入缩减为较小的反例。契约测试检查分别开发的组件之间的假设；变异测试检查测试能否发现实现中的特定改动。这些技术回答不同问题，都没有免除设计有效预期的工作。
Test Types、Coverage 与 Test Quality
Test Hierarchy 及每个层级提供什么
单元测试聚焦于较小的行为单位，可以是函数、类或几个协作对象。隔离可以指不依赖其他测试和不可控资源，并不一定要求替换所有协作者。本地定价计算就是合适的对象。运行快很有用，但称为“单元测试”不会自动使测试快速或具有确定性。
测试替身为特定目的替换依赖。桩提供指定响应，间谍记录调用，模拟对象检查预期交互，伪实现提供简化但能工作的实现。可控时钟让测试无需等待就能跨过到期边界；假的邮件发送器避免真正发送消息。替身可能与生产依赖不一致，因此不应成为集成正确性的唯一证据。
集成测试让相关组件共同运行，检查它们之间的边界。例如，数据库测试可以检查两个并发预约是否真的只能有一个提交成功。没有事务的内存伪实现不能证明这一性质。端到端测试通过系统的外部接口检查完整流程，接口可以是浏览器、API 或命令行。它把部署后的组件与配置一起检查，代价是更多准备工作和故障定位成本。
消费者驱动的契约测试记录消费者需要的交互，并用提供方验证。它可以在不运行整个应用的情况下发现不兼容的请求或响应变化。其范围是已记录的契约和提供方状态，并不会自动检查网络可达性、生产凭据、吞吐量或全部交互序列。
测试金字塔等组合建议是成本模型，而不是固定配额。局部规则的多种组合适合针对性测试；真实边界语义适合集成测试；关键业务流程适合完整流程测试。性能、安全和无障碍等问题，则可能需要跨多个层级检查。合理的组合取决于哪些失败重要，以及在哪里能可靠发现它们。
Test Coverage 及其限制
语句覆盖率记录哪些语句运行过；分支覆盖率记录哪些控制流分支运行过。覆盖率报告有助于寻找未执行的代码，但不衡量断言是否正确、需求是否完整。有些分支在有效输入下不可达，路径数量也可能随着条件与循环迅速增长。统一的百分比目标会掩盖这些差异。
对于免运费规则，假设实现误把 total >= 100 写成 total > 100。使用 99 和 101 的测试会执行条件的两种结果，也可以正确断言运费，却遗漏 100 这个缺陷点。即使分支覆盖率达到 100%，仍可能漏掉重要边界。这个例子比“只执行、不检查”的测试更进一步：有效的断言同样可能不完整。
变异测试有意修改运算符或表达式，然后重新运行测试。未被发现的修改值得调查，但它可能在有效输入范围内与原程序等价，也可能只影响无关行为。因此，变异得分取决于生成哪些变异、排除哪些变异，以及怎样处理等价情况。它是另一种诊断信息，不是实际缺陷被发现的百分比。
难以获得精确判据时，可以使用相互独立的约束。对于排序，同时检查非递减顺序和输入各元素出现次数的保留。总是返回空列表的实现满足第一个性质，却在非空输入上违反第二个。解析器与序列化器即使通过往返测试，也可能对外部格式存在同一个误解；格式规范中的示例可以提供独立检查。
Test Design Principles
选择测试数据之前，先描述需求。对于门槛规则，使用 99、100 和 101；再确定零、负数、小数和缺失值是有效输入，还是应该被拒绝。等价类与边界应同时依据契约和实现。性质测试增加生成案例和反例缩减，但仍依赖生成器能覆盖相关情况，以及所选性质确实有用。
准备所需状态，执行操作，再断言可观察的后果。一个行为可以需要多个断言：取消预约可能既要释放座位，又要恰好产生一个退款请求。除非内部调用顺序属于真实契约，否则避免断言它。测试应提供足够上下文，使失败能够指出可能违反的要求。
根据需要控制时钟、随机种子、测试数据和异步完成条件。使用明确的同步机制，不要假定任务在随意等待一段时间后必然完成。测试结束后回滚事务，只能隔离参与该事务的工作；其他进程或连接可能已经独立提交。并行测试也需要独立资源或明确协调。不稳定结果可能暴露真实竞态，应先调查，再判断是否属于测试噪声。
测试集整体的可靠性也很重要。如果 100 个测试各自独立地有 0.1% 的概率误报，那么一次运行至少出现一个误报的概率是 1 − 0.999¹⁰⁰ ≈ 9.52%。实际失败未必独立，但这一计算解释了为什么单个测试中罕见的问题会使整个测试集难以信任。一直重跑直到通过会隐藏这个信号。临时隔离测试时，应指定负责人和修复计划，并保留失败证据。
AI 生成的测试也需要同样审查。从实现照搬的测试可能重复实现的错误，包括错误的门槛。应独立提供需求和反例，检查断言，并在可行时有意运行错误版本，确认测试能够发现预定的错误。
学习这一部分会改变什么
你会学着追问：一次测试通过，究竟提供了什么证据？定价规则测试、真实数据库测试和完整结算流程支持不同主张。这有助于在合适的边界补充缺失检查，而不是无目的地增加测试数量。
可以从运费示例开始：先用 99 和 101 达到完整分支覆盖，再引入错误比较，观察这两个测试仍然通过；加入 100，缺陷才显现。随后用只有顺序性质的测试检查一个故意写错的排序程序，再加入元素出现次数的性质。这些练习把执行覆盖与测试判据的力度区分开来。
对于小型应用，选择一条重要的失败路径，例如重试期间取消操作，先说明要求的结果，再写测试。真实缺陷出现后，补充针对性的回归测试，并思考其原因是否提示了更广的性质或缺失的集成测试。保持测试集可理解：失败应促使调查，而不是被例行忽略。
资源
书籍与文本
Myers、Sandler 和 Badgett 的《The Art of Software Testing》讨论系统化设计测试案例。Beck 用例子展示 TDD 循环，Freeman 和 Pryce 发展从外向内的方法，Khorikov 讨论测试价值及其与实现细节的耦合。Osherove 的《The Art of Unit Testing》第三版由 Vladimir Khorikov 协作撰写，使用 JavaScript 示例。阅读时比较各方法的假设，而不把某种测试风格当成硬性规定。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Myers、Sandler、Badgett，The Art of Software Testing（第 3 版） | 系统化测试案例设计 | 深入 |
| Beck, Test-Driven Development: By Example | TDD 奠基文本 | 入门 |
| Khorikov, Unit Testing: Principles, Practices, and Patterns | 当代 unit testing 原则 | 实践 |
| Freeman & Pryce, Growing Object-Oriented Software, Guided by Tests | TDD outside-in approach | 深入 |
| Continuous Delivery testing chapters（§7.4 参考） | Deployment pipeline 中的 testing | 深入 |
| Osherove，Khorikov 协作，The Art of Unit Testing（第 3 版） | 使用 JavaScript 示例学习单元测试 | 入门 |
课程、论文与当前资料
阅读 Dijkstra 关于结构化程序设计的讨论，理解测试与一般正确性论证的区别。QuickCheck 论文解释性质和生成案例；Hypothesis 文档 则帮助实践输入生成与反例缩减。Pact 文档 展示如何用提供方验证消费者的预期。
| 资源 | 平台 | 类型 |
|---|---|---|
| Dijkstra，Notes on Structured Programming（1969 年手稿，1972 年出版） | 程序推理与测试的范围 | 深入 |
| Fowler, testing articles（免费） | martinfowler.com | 参考 |
| Claessen & Hughes, QuickCheck paper（免费） | Property-based testing 基础 | 深入 |
| Google Testing Blog / Tech on the Toilet（免费） | testing.googleblog.com | 辅助 |
实践、工具与当前资料
使用适合现有项目的测试运行器，再添加参数化边界案例和一个有用性质。检查覆盖率报告，并有意引入一个仅凭覆盖率无法发现的缺陷。变异工具可以自动生成其中部分改动。进行集成与浏览器测试时，控制数据和外部影响，并保留足够诊断信息以解释失败。
| 资源 | 平台 | 类型 |
|---|---|---|
| JUnit / pytest / Jest 文档 | 相应语言的测试框架 | 实践 |
| Coverage tools: JaCoCo / Coverage.py / Istanbul（免费） | Language-specific | 实践 |
| Hypothesis（Python，免费） | hypothesis.works | 实践 |
| fast-check（JavaScript，免费） | GitHub | 实践 |
| Playwright Test（免费） | playwright.dev | 实践 |
| Pact contract testing（免费） | pact.io | 实践 |
| PIT mutation testing（Java，免费） | pitest.org | 实践 |
| Stryker（JS/TS mutation testing，免费） | stryker-mutator.io | 实践 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 把覆盖率等同于正确性 | 已执行的分支仍可能缺少断言或边界案例。 | 结合需求与反例审查覆盖率。 |
| 把所有依赖都换成模拟对象 | 替身可能重复真实集成不满足的假设。 | 有目的地使用替身，并用真实组件检查重要边界。 |
| 测试偶然的实现细节 | 即使要求的行为不变，内部调整也会引起失败。 | 断言契约中的输出、状态转移和交互。 |
| 对不明失败一直重跑直到通过 | 重试可能掩盖不确定性和真实竞态。 | 保留证据、调查原因，并为临时隔离的测试指定负责人。 |
| 只测试成功示例 | 无效输入、中断、并发和恢复可能具有不同的行为。 | 依据契约、重要边界和可能的失败方式选择案例。 |