English
High-performance computing is the discipline of computation at the largest scales of resource consumption: thousands or millions of processor cores, terabytes or petabytes of memory, hundreds of accelerators, coordinated to solve problems that smaller systems cannot solve at all or cannot solve in usable time. The intellectual content is not simply “programming with more hardware.” At HPC scale, the fundamental engineering constraints shift. Moving data costs more than computing on it. Programs that do not analyze well for parallelism do not scale, regardless of how many processors are available. Hardware characteristics — cache hierarchies, vector unit widths, GPU warp schedulers, interconnect latencies — determine performance by factors of ten or more, and hiding behind software abstractions makes those factors invisible rather than irrelevant.
Three recognitions organize how the discipline is approached. First: exploiting parallelism at large scale requires different algorithms and different data structures than sequential computation, and the differences must be addressed at design time, not by annotating sequential code afterward. Second: performance at scale is determined by data movement at least as much as by computation, and this dominates engineering decisions from algorithm choice through data layout through communication pattern. Third: integrating hardware specifics, algorithmic choices, and numerical analysis is mandatory in ways that general-purpose engineering at smaller scales can avoid. The HPC engineer who cannot reason simultaneously about memory hierarchy behavior, parallel algorithm correctness, and floating-point error accumulation cannot make the right engineering decisions.
This section connects to the hardware foundations of §4.1 at one end and to the numerical and mathematical content of §2.3 (linear algebra), §2.4 (calculus), and §6.5 (scientific computing) at the other. The parallel programming models and the performance engineering discipline in this section are the bridge between what hardware can do and what scientific and engineering applications require.
Prerequisites: Computer organization and architecture (§4.1) — memory hierarchies, vector units, and hardware behavior are load-bearing. Linear algebra (§2.3) at implementation depth. Numerical analysis from Trefethen and Bau is needed as mathematical content deepens; Higham for floating-point error analysis.
From Ballistic Tables to Exascale: The Architecture of Scale
The motivation for high-performance computing predates the computers themselves. The first large-scale numerical computing projects were human computers — rooms of people performing arithmetic in parallel, each assigned a portion of a calculation — and the organization of this work invented parallel decomposition, task assignment, and result aggregation before any machine did it. The ENIAC, built at the University of Pennsylvania and operational in 1945, was designed to automate ballistic table computation for artillery — a problem whose human-parallel predecessor had employed over a hundred mathematicians. From the beginning, the motivation was problems too large for any individual or sequential process to handle in usable time.
The architecture of parallel computing developed through successive hardware generations, each with its own programming model and its own characteristic bottlenecks. Michael Flynn’s 1966 taxonomy — SISD, SIMD, MISD, MIMD — classified the space of possible parallel computer architectures. SIMD (Single Instruction Multiple Data) machines apply the same instruction to multiple data elements simultaneously, which fits vector processing and graphics. MIMD (Multiple Instruction Multiple Data) machines allow independent instruction streams on multiple processors, which fits general parallel computation. This classification, though simple, organized the diversity of parallel hardware and is still used in reasoning about modern processors, which combine SIMD vector units with MIMD multi-core execution.
The Cray-1, delivered to Los Alamos National Laboratory in 1976, defined the vector supercomputer. Its vector registers held 64 elements of 64-bit floating point; vector instructions operated on all 64 simultaneously. A program that could express its computation as sequences of vector operations — as scientific simulations often could — achieved performance far beyond what sequential processors provided. Seymour Cray’s architectural philosophy — a single very fast processor, extraordinarily careful hardware design, and a programming model that exposed the vector hardware directly — produced machines that dominated scientific computing through the 1980s.
Massively Parallel Processing (MPP) emerged in the late 1980s as an alternative architectural philosophy: instead of one very fast processor, use thousands of commodity processors connected by a fast network. The Connection Machine (1985) at Thinking Machines Corporation, the Intel Paragon (1992), and the Cray T3E (1995) exemplified this approach. The programming model was different from vector computing: each processor had its own memory, and communicating between processors required explicit message passing. The Message Passing Interface (MPI) standard, agreed upon by the parallel computing community in 1994, unified the fragmentary vendor-specific communication libraries and created a portable programming model for distributed-memory parallel programming. MPI remains the dominant model for large-scale distributed-memory computation thirty years later — one of the more stable technology choices in a field that otherwise changes rapidly.
Beowulf clusters, starting with the original 16-node cluster built at NASA Goddard in 1994 by Thomas Sterling and Donald Becker, demonstrated that commodity Intel processors connected by commodity Ethernet could achieve supercomputer-level performance at a fraction of the cost of purpose-built MPP machines. This democratized HPC: research groups that could not afford million-dollar systems could build cluster computers from standard PC components. The rise of Beowulf clusters shifted the center of gravity of HPC from proprietary hardware to Linux clusters, and the Linux cluster running open-source MPI implementations became the dominant HPC platform by the early 2000s.
GPU computing arrived from an unexpected direction. Graphics processing units were specialized processors designed for the parallel, arithmetic-intensive computations of 3D rendering. By the mid-2000s, GPUs had more transistors, higher memory bandwidth, and more floating-point throughput than CPUs of the same generation, and researchers were finding ways to express scientific computations as graphics operations to exploit this. NVIDIA’s CUDA platform, released in 2007, provided a general-purpose programming model for GPU computation that did not require expressing programs as graphics. CUDA made GPU programming accessible to scientists and engineers who did not know computer graphics, and the scientific computing community rapidly discovered that many HPC workloads — molecular dynamics, climate simulation, computational fluid dynamics — mapped naturally onto the GPU’s data-parallel execution model. The first Gordon Bell Prize winner using GPU acceleration came in 2009; by 2012, the top-performing supercomputer (Titan, at Oak Ridge) was GPU-accelerated.
The petaflop barrier — one quadrillion floating-point operations per second — was crossed in 2008 by the IBM Roadrunner at Los Alamos. The exaflop barrier — one quintillion floating-point operations per second — was crossed in 2022 by the Frontier system at Oak Ridge, using AMD CPUs and GPUs. Each crossing required solving engineering problems that did not exist at the previous scale: programming models for heterogeneous CPU-GPU systems, interconnects that could sustain communication between hundreds of thousands of devices, file systems that could handle the I/O demands of the applications, and power and cooling infrastructure at power levels measured in megawatts.
The convergence of HPC and machine learning that began around 2016 has fundamentally changed the character of HPC investment. Training large language models requires tens of thousands of GPU accelerators running for weeks, coordinated through sophisticated distributed training strategies — tensor parallelism, pipeline parallelism, data parallelism — that are direct descendants of HPC parallel programming techniques. The engineering challenges — communication bottlenecks between GPUs, memory capacity limitations per device, numerical precision management during training — are HPC engineering challenges applied to ML workloads. NVIDIA’s H100 and H200 GPUs, Intel’s Gaudi accelerators, and Google’s TPU v4 and v5 are designed as much for ML training as for traditional scientific computing. The largest HPC systems being built as of 2025 — El Capitan at Lawrence Livermore, the NSF Leadership computing platforms — reflect both missions: scientific simulation and AI training. The disciplines have converged to the point where an engineer trained in HPC parallel programming and performance analysis can move between scientific and ML workloads without re-learning the fundamental skills.
Parallelism, Data Movement, and Hardware at Scale
Extracting and Managing Parallelism
A program runs faster on parallel hardware only to the extent that its work decomposes into pieces that can execute simultaneously on different processing units. Finding this decomposition — and expressing it in a form the hardware can exploit — is the discipline’s central activity.
Amdahl’s law captures the fundamental constraint: if a fraction f of a program is inherently sequential, the maximum speedup achievable with infinite parallelism is 1/(1-f). A program that is 95% parallel and 5% sequential cannot be made faster than 20× even with unlimited processors. Gustafson’s law provides the complementary view for scaled workloads: if problem size grows with the number of processors (a fixed problem per processor), the sequential fraction becomes a decreasing proportion of the total work, and the parallel portion can grow without bound. The practical implication is that parallelism pays most for workloads where the problem size scales with available hardware — which characterizes most scientific simulation and most ML training.
The work-span model provides finer-grained analysis. The work W of a parallel computation is the total number of operations; the span S is the length of the critical path — the longest chain of sequential dependencies. With P processors, the runtime is at least max(W/P, S). A computation with large span cannot be parallelized beyond 1/S efficiency; reducing span requires either restructuring the algorithm or accepting additional work to expose more parallelism. Many parallel algorithm design choices trade additional work for reduced span — sorting networks accept more comparisons than comparison sorts but have lower span.
The parallel programming models differ in what parallelism they expose and what it costs. Shared-memory parallelism (OpenMP) allows multiple threads to access a common address space, communicating through shared variables. This is natural for programs where the data fits in shared memory and thread synchronization costs are manageable. The correctness risks — data races from unsynchronized access to shared state — are real and the debugging tools (ThreadSanitizer) are imperfect. Distributed-memory parallelism (MPI) requires explicit message passing between processes with separate address spaces. There are no data races; there are deadlocks, incorrect message matching, and the constant complexity of data partitioning. GPU parallelism (CUDA, ROCm) executes thousands of lightweight threads simultaneously in SIMT (Single Instruction Multiple Thread) fashion, with warp-level synchronization and memory access patterns that determine whether the GPU’s bandwidth is fully utilized.
Each model has its domain: shared-memory parallelism is appropriate for problems that fit in a node’s memory and have irregular access patterns that benefit from shared address space; distributed memory is necessary when the problem exceeds a single node; GPU parallelism is appropriate for data-parallel kernels with high arithmetic intensity and regular memory access. Real HPC applications frequently combine all three — MPI between nodes, OpenMP or GPU acceleration within each node.
Data Movement: The True Bottleneck
At HPC scale, moving data costs more than computing on it. A 64-bit floating-point multiply takes approximately one nanosecond on a modern processor; reading the same value from DRAM takes 50-100 nanoseconds; reading it from another node over InfiniBand takes microseconds. Programs that compute rapidly but move data frequently are bandwidth-limited; improving the arithmetic performance of a bandwidth-limited program produces no improvement.
The Roofline model makes this quantitative. For any computational kernel, the arithmetic intensity is the ratio of floating-point operations to bytes of memory traffic. The Roofline model plots performance (in FLOPS/s) against arithmetic intensity (FLOPS/byte). A horizontal ceiling gives the processor’s peak FLOP rate; a diagonal ceiling gives the memory bandwidth limit (FLOPS/s = bandwidth × arithmetic intensity). Every kernel falls either under the peak FLOP ceiling (compute-limited) or under the memory bandwidth ceiling (bandwidth-limited). The model identifies immediately which bound applies and what optimization can achieve: a compute-limited kernel benefits from algorithmic improvements that reduce FLOP count; a bandwidth-limited kernel benefits from data layout changes, cache-aware algorithms, or loop transformations that improve data reuse.
Cache-aware algorithms structure computation to maximize data reuse within the cache before moving to the next data. Matrix multiplication illustrates the principle: a naive triple-loop implementation moves each element of the matrix multiple times for each arithmetic operation; a blocked (tiled) implementation processes small blocks that fit in cache, performing many arithmetic operations on each block before moving to the next. The difference in performance can be a factor of ten or more on large matrices, purely from better cache utilization with identical arithmetic.
Communication-avoiding algorithms take this further: they restructure algorithms to perform additional computation in exchange for reduced communication. The standard matrix multiplication on a distributed memory system requires communicating the entire matrix multiple times; the SUMMA and Cannon’s algorithms reduce communication volume; the 2.5D and 3D matrix multiplication algorithms (Solomonik and Demmel) reduce it further by using additional memory to store copies of data. In communication-dominated regimes — large matrices distributed across many nodes — these algorithms outperform the standard algorithm despite requiring more total arithmetic.
Floating-Point and Numerical Correctness
Floating-point arithmetic is not real arithmetic. At HPC scale, small per-operation errors compound over millions or billions of operations, and programs that produce plausible-looking but numerically incorrect results are as wrong as programs with bugs — more dangerous, because the error is harder to detect.
The condition number of a problem measures how sensitive its solution is to small perturbations in the input. A poorly conditioned problem amplifies small floating-point errors into large output errors, regardless of how carefully the algorithm is implemented. Identifying the condition number of a computation before applying it to data is part of the competence of numerical HPC programming. Gaussian elimination with partial pivoting has favorable condition behavior for most matrices; without pivoting, it is numerically unstable even for well-conditioned matrices.
Parallel floating-point summation is not associative: the order in which terms are summed determines the rounding errors, and parallel reductions aggregate partial sums in an order determined by the scheduling of parallel tasks, which can change between runs. This produces non-reproducible results — different runs of the same parallel program on the same input can produce different output. For scientific computing, where results must be reproducible and validated against known solutions, this is a genuine problem. Compensated summation algorithms (Kahan summation), sorted summation, and reproducible floating-point reduction algorithms trade performance for reproducibility; choosing the right tradeoff requires understanding the specific numerical requirements of the application.
What Studying This Changes
HPC changes how practitioners reason about computation as a physical activity rather than an abstract one.
The first change is reasoning about performance at the hardware level rather than the algorithmic level alone. Asymptotic complexity is necessary but insufficient. A program’s actual performance on specific hardware is determined by its memory access patterns, its instruction-level parallelism, its vectorization opportunities, and its communication overhead. The Roofline model, hardware performance counters, and empirical profiling are the tools for this analysis. Without them, performance optimization is guessing; with them, it is engineering.
The second change is designing for parallelism as a primary concern. Retrofitting parallelism onto sequential programs typically fails because the data structures and algorithms were designed around sequential access patterns. HPC thinking considers parallel decomposition first, chooses data structures that support the decomposition, and analyzes locality implications before implementation. This discipline produces programs that scale — not because they use the right parallel primitives, but because their structure is compatible with large-scale parallel execution.
The third change is numerical discipline as a routine engineering concern. The difference between numerically stable and unstable algorithms is practically significant and not always obvious. Practitioners who have studied numerical analysis check condition numbers, test algorithms against known solutions, and choose numerically stable implementations even when unstable alternatives appear computationally cheaper. This discipline prevents the class of failures — results that look right but are numerically wrong — that has produced erroneous scientific conclusions and has no other systematic remedy.
Resources
Books and Texts
Pacheco and Malensek’s An Introduction to Parallel Programming (2nd ed., Morgan Kaufmann, 2021) is the standard entry to parallel programming with MPI, OpenMP, and CUDA. Its approach — worked examples, explicit correctness analysis, and performance analysis alongside each programming model — reflects the integration of programming, correctness, and performance that HPC requires. It is appropriate as the primary text for learners engaging with parallel programming systematically.
Hager and Wellein’s Introduction to High Performance Computing for Scientists and Engineers (CRC Press, 2010) covers the architecture, performance analysis, and programming discipline of HPC with more emphasis on the engineering perspective than Pacheco. Its treatment of the Roofline model, memory hierarchy effects, and the integration of algorithmic and hardware considerations is the best available introduction to HPC performance engineering. The 2010 date is a limitation — it predates GPU dominance — but the core performance engineering content remains current.
Kirk and Hwu’s Programming Massively Parallel Processors (4th ed., Morgan Kaufmann, 2022) is the academic standard for GPU programming with CUDA. It covers the CUDA programming model, GPU architecture (from a programmer’s perspective), memory hierarchy management, warp-level execution, and performance optimization systematically. The CUDA Programming Guide (free at developer.nvidia.com) is the authoritative reference; Kirk and Hwu is the pedagogical treatment.
For numerical foundations, Trefethen and Bau’s Numerical Linear Algebra (SIAM, 1997) is the most readable rigorous treatment of the numerical linear algebra that underlies most scientific computing. Demmel’s Applied Numerical Linear Algebra (SIAM, 1997) goes deeper on implementation concerns. Higham’s Accuracy and Stability of Numerical Algorithms (2nd ed., SIAM, 2002) is the definitive reference on floating-point error analysis — the treatment that makes it possible to reason rigorously about whether a computed result is correct.
Gropp, Lusk, and Skjellum’s Using MPI (3rd ed., MIT Press, 2014) is the standard MPI text. The MPI 4.0 standard (free at mpi-forum.org) is the authoritative specification. For parallel algorithms at depth, JaJa’s An Introduction to Parallel Algorithms (Addison-Wesley, 1992) provides the foundational theoretical treatment.
| Book | Role | Type |
|---|---|---|
| Pacheco & Malensek, An Introduction to Parallel Programming (2nd ed.) | Standard parallel programming entry | Entry |
| Hager & Wellein, Introduction to HPC for Scientists and Engineers | HPC performance engineering entry | Entry |
| Kirk & Hwu, Programming Massively Parallel Processors (4th ed.) | GPU programming academic standard | Entry |
| CUDA Programming Guide (free, NVIDIA) | Authoritative GPU programming reference | Reference |
| Trefethen & Bau, Numerical Linear Algebra | Numerical linear algebra foundation | Reference |
| Demmel, Applied Numerical Linear Algebra | Numerical LA at implementation depth | Depth |
| Higham, Accuracy and Stability of Numerical Algorithms (2nd ed.) | Floating-point error analysis | Reference |
| Gropp, Lusk & Skjellum, Using MPI (3rd ed.) | MPI standard text | Reference |
| JaJa, An Introduction to Parallel Algorithms | Parallel algorithms foundations | Depth |
| Hennessy & Patterson, Computer Architecture: Quantitative Approach (see §4.1) | Architecture reference for HPC | Reference |
Courses and Lectures
MIT 6.172 (Performance Engineering of Software Systems, free on MIT OCW) covers the full stack of performance engineering — compiler optimization, memory hierarchy, parallel programming, and performance measurement — in a single rigorous course. The assignments require optimizing real programs using profiling-guided methodology, and the course enforces the discipline of measurement before optimization that distinguishes effective performance work from guessing.
Caltech CS 179 (GPU Programming with CUDA, free lecture materials) provides structured GPU programming education. NVIDIA’s CUDA samples and CUDA developer blog provide working examples of CUDA optimization techniques with explanations.
SC (Supercomputing Conference) tutorials (free, recorded and posted annually) cover current HPC topics — programming models, performance analysis, emerging hardware — with a depth that matches the conference’s practitioner audience. The tutorials on communication-avoiding algorithms, performance portability frameworks, and exascale programming models are particularly valuable.
| Course | Platform | Type |
|---|---|---|
| MIT 6.172 Performance Engineering (free) | MIT OCW / YouTube | Entry |
| Caltech CS 179 GPU Programming (free) | Caltech course site | Entry |
| Supercomputing (SC) tutorials (free) | SC archive / YouTube | Reference |
| LLNL HPC Tutorials (free) | hpc-tutorials.llnl.gov | Entry |
| HPC Carpentry lessons (free) | hpc-carpentry.org | Practice |
Practice, Tools, and Current Sources
NVIDIA Nsight (free) is the standard GPU profiling and debugging environment. Profiling a CUDA kernel, examining memory access patterns, and identifying warp divergence and occupancy limitations is the practical work of GPU optimization. perf (Linux) and Intel VTune (Intel, free for students) provide CPU profiling with hardware performance counters that make memory hierarchy behavior visible.
The Williams-Waterman-Patterson Roofline paper (free, Communications of the ACM 2009) is the essential performance modeling tool — short, readable, and directly applicable. Applying the Roofline model to actual kernels, by measuring their arithmetic intensity with profiling tools and plotting against measured hardware ceilings, is the primary performance analysis methodology.
LINPACK benchmark and its successors (HPL, HPCG) are the standard metrics for supercomputer performance and are documented free at hpcg-benchmark.org and netlib.org. Understanding what LINPACK measures (dense matrix factorization, highly regular, cache-friendly) and what it does not measure (real application performance, irregular access, memory-bandwidth dominance) is necessary context for interpreting Top500 rankings.
The BLAS (Basic Linear Algebra Subprograms) and LAPACK libraries are the foundational numerical software that HPC builds on. Understanding their API and using them appropriately — rather than reimplementing basic linear algebra operations — is baseline HPC competence. OpenBLAS, MKL, and cuBLAS are the standard implementations for CPU and GPU platforms.
| Resource | Platform | Type |
|---|---|---|
| NVIDIA Nsight (free) | developer.nvidia.com | Practice |
| perf / Intel VTune (free) | Linux / intel.com | Practice |
| Roofline model paper (free) | CACM 2009 | Depth |
| BLAS / LAPACK / OpenBLAS (free) | netlib.org / github.com/xianyi/OpenBLAS | Practice |
| OpenMP reference guides and tutorials (free) | openmp.org | Reference |
| Top500 / HPCG benchmark (free) | top500.org / hpcg-benchmark.org | Reference |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Parallelism as annotation | Adding #pragma omp parallel for or CUDA kernel launches to existing sequential code treats parallelism as syntactic transformation rather than algorithm design. The result is nominally parallel code that fails to scale: data dependencies prevent useful parallelism, synchronization overhead dominates the speedup, sequential bottlenecks limit Amdahl scaling. |
Treat parallelism as an algorithm design concern from the start. Study parallel algorithms to understand which computations parallelize well and which require rethinking. Design data structures for parallel access patterns before writing code. Analyze the computation’s work and span before choosing a parallel decomposition. |
| Speedup relative to an unoptimized baseline | Reporting GPU speedup over an unoptimized sequential CPU implementation is misleading. An optimized CPU implementation using SIMD, cache-friendly access, and compiler auto-vectorization may close most of the apparent gap. A claimed 50× speedup against naive sequential code frequently becomes 5× against optimized sequential code. | Always measure against the best available sequential baseline. Profile the CPU implementation with vectorization enabled and cache behavior optimized before concluding that GPU offload is warranted. The decision to use GPU acceleration should rest on comparison with well-optimized sequential code. |
| Ignoring data movement | Focusing on computational intensity while neglecting data movement produces algorithms with impressive FLOP counts that perform poorly in practice. Most HPC kernels are bandwidth-limited rather than compute-limited, and optimizing the computation portion of a bandwidth-limited program produces no improvement. | Apply the Roofline model before optimizing. Profile memory bandwidth utilization alongside computation. Prioritize cache-aware algorithm design, data layout choices, and communication minimization. Accept additional computation in exchange for reduced data movement where arithmetic intensity analysis justifies it. |
| GPU as uniform speedup | GPU acceleration is marketed as a general performance multiplier. Code that ignores warp-level execution, produces divergent branches, uses non-coalesced memory access patterns, or underoccupies the device will run slower than well-optimized CPU code despite using more powerful hardware. | Engage with GPU architecture before writing GPU code. Understand warp execution, memory coalescing, shared memory, and occupancy. Profile with Nsight to understand where time is actually spent. GPU optimization is its own engineering discipline and cannot be acquired by learning the CUDA API alone. |
| Treating floating-point as real arithmetic | At HPC scale, small per-operation floating-point errors compound over millions of operations. Cancellation errors, accumulated rounding, and order-of-operations dependence can produce results that look plausible but are numerically wrong. For scientific computing, numerically incorrect results are as bad as incorrect code — and harder to detect. | Study numerical analysis alongside HPC programming. Understand floating-point error propagation, condition numbers, and the difference between numerically stable and unstable algorithms. Test numerical results against known solutions or analytical bounds. Read Higham on floating-point analysis for the applications where precision matters. |
| Optimizing without profiling | The most common form of ineffective HPC optimization is optimizing the wrong bottleneck. Engineers assume that computation is the bottleneck when it is actually memory bandwidth; assume that computation on one device is the bottleneck when it is actually communication between devices; assume that a specific loop is hot when the profiler shows that it is not. | Profile before optimizing, every time. Use hardware performance counters (Nsight, perf, VTune) to identify the actual bottleneck. Form a hypothesis about the bottleneck, measure to test the hypothesis, then optimize. The discipline of measurement-before-optimization produces a 10× improvement in the effectiveness of optimization effort compared to intuition-based optimization. |
中文
高性能计算是在最大规模资源消耗上进行计算的学科：成千上万乃至数百万个 processor cores，TB 或 PB 级内存，数百个 accelerators，被协调起来解决那些较小系统根本无法解决、或无法在可用时间内解决的问题。它的智识内容并不只是“用更多硬件编程”。在 HPC 规模下，基本工程约束会发生转移。移动数据的成本高于在数据上计算。不能被良好并行分析的程序，无论有多少 processors 可用，都不会扩展。硬件特征——cache hierarchies、vector unit widths、GPU warp schedulers、interconnect latencies——会以十倍甚至更高倍数决定性能；躲在软件抽象背后，只会让这些因素不可见，而不是让它们变得无关。
这门学科的学习由三个认识组织起来。第一：在大规模上利用 parallelism，需要不同于 sequential computation 的算法和数据结构，而且这些差异必须在设计阶段处理，而不是事后给顺序代码加注解。第二：大规模性能至少和 computation 一样由 data movement 决定；这一点会支配从算法选择、数据布局到通信模式的工程决策。第三：必须同时整合硬件细节、算法选择和数值分析，而这是小规模通用工程可以暂时回避的。一个 HPC engineer 如果不能同时推理 memory hierarchy behavior、parallel algorithm correctness 和 floating-point error accumulation，就无法做出正确工程决策。
本节一端连接 §4.1 的硬件基础，另一端连接 §2.3（线性代数）、§2.4（微积分）和 §6.5（科学计算）的数值与数学内容。本节中的 parallel programming models 和 performance engineering discipline，是硬件能力与科学、工程应用需求之间的桥梁。
前置知识：计算机组成与体系结构（§4.1）——memory hierarchies、vector units 和 hardware behavior 是承重内容。实现深度的线性代数（§2.3）。随着数学内容加深，需要 Trefethen 和 Bau 的数值分析；浮点误差分析则需要 Higham。
从弹道表到 Exascale：规模的架构
高性能计算的动机早于计算机本身。最早的大规模数值计算项目是 human computers——一屋子人并行执行算术，每个人负责计算的一部分——这种工作组织方式在机器出现之前，就已经发明了 parallel decomposition、task assignment 和 result aggregation。ENIAC 由 University of Pennsylvania 建造，于 1945 年投入运行，设计目的正是自动化炮兵弹道表计算——此前这个问题的人类并行版本曾雇用超过一百名数学家。从一开始，动机就是处理那些任何个人或顺序过程都无法在可用时间内处理的问题。
并行计算架构经过连续几代硬件发展，每一代都有自己的 programming model 和典型瓶颈。Michael Flynn 1966 年的分类法——SISD、SIMD、MISD、MIMD——对可能的并行计算机架构空间进行了分类。SIMD（Single Instruction Multiple Data）machines 同时把同一条指令应用到多个数据元素上，适合 vector processing 和 graphics。MIMD（Multiple Instruction Multiple Data）machines 允许多个 processors 上的独立指令流，适合通用并行计算。这个分类虽然简单，却组织了并行硬件的多样性；它至今仍用于推理现代处理器，因为现代处理器结合了 SIMD vector units 和 MIMD multi-core execution。
Cray-1 于 1976 年交付给 Los Alamos National Laboratory，定义了 vector supercomputer。它的 vector registers 保存 64 个 64-bit floating point elements；vector instructions 会同时作用于全部 64 个元素。一个程序如果能把计算表达为 vector operations 序列——科学仿真经常可以这样做——就能获得远超顺序处理器的性能。Seymour Cray 的架构哲学——一个非常快的处理器、极其精心的硬件设计，以及直接暴露 vector hardware 的 programming model——制造出的机器主导了 1980 年代的科学计算。
Massively Parallel Processing（MPP）在 1980 年代后期作为另一种架构哲学出现：不用一个非常快的处理器，而是使用数千个由高速网络连接的 commodity processors。Thinking Machines Corporation 的 Connection Machine（1985）、Intel Paragon（1992）和 Cray T3E（1995）都是这一方向的例子。其 programming model 不同于 vector computing：每个 processor 有自己的内存，processors 之间通信需要显式 message passing。Message Passing Interface（MPI）standard 由并行计算共同体于 1994 年达成一致，它统一了碎片化的 vendor-specific communication libraries，并为 distributed-memory parallel programming 创造了可移植 programming model。三十年后的今天，MPI 仍然是大规模 distributed-memory computation 的主导模型——在一个变化通常很快的领域中，这是相当稳定的技术选择。
Beowulf clusters 始于 Thomas Sterling 和 Donald Becker 于 1994 年在 NASA Goddard 构建的原始 16-node cluster。它证明，使用 commodity Ethernet 连接的 commodity Intel processors，可以以专用 MPP machines 一小部分成本实现 supercomputer-level performance。这使 HPC 民主化：买不起百万美元系统的研究组，可以用标准 PC 组件搭建 cluster computers。Beowulf clusters 的兴起，把 HPC 的重心从专有硬件转向 Linux clusters；到 2000 年代早期，运行开源 MPI implementations 的 Linux cluster 已经成为主导 HPC platform。
GPU computing 来自一个意想不到的方向。Graphics processing units 原本是为 3D rendering 中并行、算术密集计算设计的专用处理器。到 2000 年代中期，GPUs 相比同代 CPUs 拥有更多 transistors、更高 memory bandwidth 和更高 floating-point throughput；研究者也开始寻找把科学计算表达为 graphics operations 的方法，以利用这些能力。NVIDIA 于 2007 年发布 CUDA platform，为 GPU computation 提供通用 programming model，不再要求把程序表达成 graphics。CUDA 使不懂计算机图形学的科学家和工程师也能接触 GPU programming；科学计算共同体很快发现，许多 HPC workloads——molecular dynamics、climate simulation、computational fluid dynamics——自然适配 GPU 的 data-parallel execution model。第一个使用 GPU acceleration 获得 Gordon Bell Prize 的项目出现在 2009 年；到 2012 年，性能最高的 supercomputer（Oak Ridge 的 Titan）已经使用 GPU acceleration。
Petaflop barrier——每秒一千万亿次 floating-point operations——由 Los Alamos 的 IBM Roadrunner 于 2008 年跨越。Exaflop barrier——每秒一百京次 floating-point operations——由 Oak Ridge 的 Frontier system 于 2022 年跨越，使用 AMD CPUs 和 GPUs。每一次跨越都要求解决前一规模不存在的工程问题：heterogeneous CPU-GPU systems 的 programming models，能够支撑数十万个 devices 之间通信的 interconnects，能够处理应用 I/O 需求的 file systems，以及以 megawatts 计的电力和冷却基础设施。
大约从 2016 年开始的 HPC 与 machine learning 融合，已经根本改变了 HPC 投资的性质。训练 large language models 需要数万块 GPU accelerators 连续运行数周，并通过复杂 distributed training strategies 协调——tensor parallelism、pipeline parallelism、data parallelism——这些都是 HPC parallel programming techniques 的直接后代。工程挑战——GPUs 之间的通信瓶颈、每个 device 的 memory capacity limitations、训练期间的 numerical precision management——都是应用到 ML workloads 上的 HPC engineering challenges。NVIDIA H100 和 H200 GPUs、Intel Gaudi accelerators，以及 Google TPU v4 和 v5，既为 ML training 设计，也为传统 scientific computing 设计。截至 2025 年，正在建设的最大 HPC systems——Lawrence Livermore 的 El Capitan、NSF Leadership computing platforms——同时反映了这两种任务：科学仿真和 AI training。这两门学科已经融合到这样一种程度：一个接受过 HPC parallel programming 和 performance analysis 训练的工程师，可以在 scientific 和 ML workloads 之间移动，而不必重新学习基础技能。
Parallelism、Data Movement 与 Scale 下的硬件
提取并管理 Parallelism
一个程序只有在其工作可以分解为能在不同 processing units 上同时执行的部分时，才能在并行硬件上运行得更快。找到这种分解，并以硬件可利用的形式表达它，是这门学科的中心活动。
Amdahl’s law 捕捉了根本约束：如果程序中有 fraction f 是天然顺序的，那么即使拥有无限 parallelism，最大 speedup 也只能是 1/(1-f)。一个 95% 可并行、5% 顺序的程序，即使用无限 processors，也不可能快过 20×。Gustafson’s law 为 scaled workloads 提供互补视角：如果问题规模随 processors 数量增长（每个 processor 处理固定问题量），顺序部分在总工作中的比例会下降，并行部分则可以无界增长。实践含义是：parallelism 对那些问题规模能随可用硬件扩展的 workloads 最有价值——这正是多数 scientific simulation 和多数 ML training 的特征。
Work-span model 提供更细粒度分析。并行计算的 work W 是总操作数；span S 是 critical path 的长度——也就是最长 sequential dependencies 链。使用 P 个 processors 时，runtime 至少是 max(W/P, S)。Span 很大的计算无法超过 1/S efficiency 地并行化；减少 span 需要重构算法，或接受额外 work 来暴露更多 parallelism。许多 parallel algorithm design choices 都是在用额外 work 换取更低 span——sorting networks 比 comparison sorts 需要更多 comparisons，但 span 更低。
Parallel programming models 的差异在于它们暴露什么 parallelism，以及这种暴露有什么成本。Shared-memory parallelism（OpenMP）允许多个 threads 访问共同 address space，并通过 shared variables 通信。这适合数据能放入共享内存、且 thread synchronization costs 可控的程序。正确性风险——对 shared state 的未同步访问导致 data races——是真实的，而调试工具（ThreadSanitizer）并不完美。Distributed-memory parallelism（MPI）要求在具有独立 address spaces 的 processes 之间显式 message passing。这里没有 data races；但有 deadlocks、incorrect message matching，以及 data partitioning 的持续复杂性。GPU parallelism（CUDA、ROCm）以 SIMT（Single Instruction Multiple Thread）方式同时执行数千个 lightweight threads，其 warp-level synchronization 和 memory access patterns 决定 GPU 的 bandwidth 是否被充分利用。
每种模型都有自己的适用领域：shared-memory parallelism 适合能放入单个 node 内存、且具有 irregular access patterns、能从 shared address space 受益的问题；distributed memory 在问题超过单个 node 时是必要的；GPU parallelism 适合 arithmetic intensity 高、memory access 规则的 data-parallel kernels。真实 HPC applications 经常把三者结合起来——nodes 之间用 MPI，每个 node 内部用 OpenMP 或 GPU acceleration。
Data Movement：真正的瓶颈
在 HPC 规模下，移动数据比在数据上计算更昂贵。在现代处理器上，一次 64-bit floating-point multiply 大约需要一纳秒；从 DRAM 读取同一个值需要 50–100 纳秒；从另一个 node 通过 InfiniBand 读取则需要微秒级时间。那些计算很快但频繁移动数据的程序是 bandwidth-limited；提高 bandwidth-limited program 的算术性能不会带来任何改进。
Roofline model 使这一点可量化。对任何 computational kernel，arithmetic intensity 是 floating-point operations 与 memory traffic bytes 的比例。Roofline model 把 performance（FLOPS/s）画在 arithmetic intensity（FLOPS/byte）之上。水平 ceiling 表示 processor 的 peak FLOP rate；对角 ceiling 表示 memory bandwidth limit（FLOPS/s = bandwidth × arithmetic intensity）。每个 kernel 要么落在 peak FLOP ceiling 之下（compute-limited），要么落在 memory bandwidth ceiling 之下（bandwidth-limited）。这个模型可以立刻识别哪种 bound 生效，以及优化可以达到什么效果：compute-limited kernel 受益于减少 FLOP count 的算法改进；bandwidth-limited kernel 受益于 data layout changes、cache-aware algorithms 或改善 data reuse 的 loop transformations。
Cache-aware algorithms 会组织计算，使数据在进入下一批数据前尽可能在 cache 中被复用。矩阵乘法展示了这个原则：朴素 triple-loop 实现会让每个 matrix element 在每次算术操作中被移动多次；blocked（tiled）实现则处理能放入 cache 的小 blocks，在移动到下一个 block 前，对每个 block 执行大量算术操作。对大型矩阵而言，二者性能差异可能达到十倍甚至更多，而算术本身完全相同，差异纯粹来自更好的 cache utilization。
Communication-avoiding algorithms 更进一步：它们通过重构算法，用额外计算换取更少通信。Distributed memory system 上的标准矩阵乘法需要多次通信整个 matrix；SUMMA 和 Cannon’s algorithms 减少通信量；2.5D 和 3D matrix multiplication algorithms（Solomonik 和 Demmel）通过使用额外内存存储数据副本进一步降低通信量。在 communication-dominated regimes 中——大型矩阵分布在许多 nodes 上——这些算法虽然需要更多总算术量，却会超过标准算法。
Floating-Point 与数值正确性
Floating-point arithmetic 不是真实算术。在 HPC 规模下，每次操作的小误差会在数百万或数十亿次操作中累积；产生看似合理但数值错误结果的程序，和有 bug 的程序一样错误——而且更危险，因为错误更难检测。
一个问题的 condition number 衡量其解对输入微小扰动的敏感程度。Poorly conditioned problem 会把小 floating-point errors 放大成大 output errors，无论算法实现多么谨慎。在把计算应用到数据之前识别 condition number，是 numerical HPC programming 能力的一部分。Gaussian elimination with partial pivoting 对多数 matrices 有良好的 condition behavior；如果没有 pivoting，即使对 well-conditioned matrices，它也是数值不稳定的。
Parallel floating-point summation 不满足 associative：terms 被求和的顺序会决定 rounding errors，而 parallel reductions 会以 parallel tasks 调度决定的顺序聚合 partial sums，这个顺序可能在不同运行之间改变。这会产生 non-reproducible results——同一个 parallel program 在同一个 input 上多次运行，可能产生不同 output。对于 scientific computing，这是真实问题，因为结果必须可复现，并且要用 known solutions 验证。Compensated summation algorithms（Kahan summation）、sorted summation 和 reproducible floating-point reduction algorithms 会用性能换取 reproducibility；选择正确权衡，需要理解应用的具体数值要求。
学习这一部分会改变什么
HPC 会改变实践者理解计算的方式：计算不是抽象活动，而是物理活动。
第一个变化，是在硬件层面而不只是算法层面推理性能。Asymptotic complexity 是必要的，但不充分。一个程序在具体硬件上的真实性能，由 memory access patterns、instruction-level parallelism、vectorization opportunities 和 communication overhead 决定。Roofline model、hardware performance counters 和 empirical profiling 是这种分析的工具。没有它们，性能优化就是猜测；有了它们，性能优化才是工程。
第二个变化，是把 parallelism 作为一等关切来设计。把 parallelism 事后补到 sequential programs 上通常会失败，因为 data structures 和 algorithms 是围绕顺序访问模式设计的。HPC thinking 会先考虑 parallel decomposition，选择支持这种分解的数据结构，并在实现前分析 locality implications。这种纪律产生可扩展程序——不是因为它们使用了正确 parallel primitives，而是因为它们的结构与 large-scale parallel execution 兼容。
第三个变化，是把数值纪律作为常规工程关切。Numerically stable 和 unstable algorithms 之间的差异具有实践意义，而且并不总是显而易见。学习过 numerical analysis 的实践者会检查 condition numbers，用 known solutions 测试算法，并选择数值稳定实现，即使不稳定替代方案看起来计算成本更低。这种纪律可以防止一类失败——结果看起来正确但数值上错误——这类失败曾导致错误科学结论，也没有其他系统性补救方案。
资源
书籍与文本
Pacheco 和 Malensek 的 An Introduction to Parallel Programming（第 2 版，Morgan Kaufmann，2021）是 MPI、OpenMP 和 CUDA 并行编程的标准入口。它的方法——worked examples、显式 correctness analysis，并把 performance analysis 与每种 programming model 放在一起——反映了 HPC 所要求的编程、正确性和性能整合。对于系统性接触并行编程的学习者，它适合作为主教材。
Hager 和 Wellein 的 Introduction to High Performance Computing for Scientists and Engineers（CRC Press，2010）覆盖 HPC 的架构、性能分析和编程纪律，比 Pacheco 更强调工程视角。它对 Roofline model、memory hierarchy effects，以及算法与硬件因素整合的处理，是目前最好的 HPC performance engineering 入门。2010 年这个时间是一个限制——它早于 GPU 主导时代——但其核心性能工程内容仍然有效。
Kirk 和 Hwu 的 Programming Massively Parallel Processors（第 4 版，Morgan Kaufmann，2022）是 CUDA GPU programming 的学术标准。它系统覆盖 CUDA programming model、从程序员视角看的 GPU architecture、memory hierarchy management、warp-level execution 和 performance optimization。CUDA Programming Guide（可在 developer.nvidia.com 免费获取）是权威参考；Kirk 和 Hwu 则是教学处理。
对于数值基础，Trefethen 和 Bau 的 Numerical Linear Algebra（SIAM，1997）是支撑大多数科学计算的 numerical linear algebra 中最易读的严格处理。Demmel 的 Applied Numerical Linear Algebra（SIAM，1997）在实现关切上更深入。Higham 的 Accuracy and Stability of Numerical Algorithms（第 2 版，SIAM，2002）是 floating-point error analysis 的权威参考——它提供了严谨推理计算结果是否正确所需的处理。
Gropp、Lusk 和 Skjellum 的 Using MPI（第 3 版，MIT Press，2014）是标准 MPI 文本。MPI 4.0 standard（可在 mpi-forum.org 免费获取）是权威 specification。对于深入 parallel algorithms，JaJa 的 An Introduction to Parallel Algorithms（Addison-Wesley，1992）提供了基础理论处理。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Pacheco & Malensek, An Introduction to Parallel Programming（第 2 版） | 标准并行编程入口 | 入门 |
| Hager & Wellein, Introduction to HPC for Scientists and Engineers | HPC performance engineering 入口 | 入门 |
| Kirk & Hwu, Programming Massively Parallel Processors（第 4 版） | GPU programming 学术标准 | 入门 |
| CUDA Programming Guide（免费，NVIDIA） | 权威 GPU programming 参考 | 参考 |
| Trefethen & Bau, Numerical Linear Algebra | Numerical linear algebra 基础 | 参考 |
| Demmel, Applied Numerical Linear Algebra | 实现深度的 numerical LA | 深入 |
| Higham, Accuracy and Stability of Numerical Algorithms（第 2 版） | Floating-point error analysis | 参考 |
| Gropp, Lusk & Skjellum, Using MPI（第 3 版） | MPI 标准文本 | 参考 |
| JaJa, An Introduction to Parallel Algorithms | Parallel algorithms 基础 | 深入 |
| Hennessy & Patterson, Computer Architecture: Quantitative Approach（见 §4.1） | HPC 所需体系结构参考 | 参考 |
课程与讲座
MIT 6.172（Performance Engineering of Software Systems，MIT OCW 免费）用一门严谨课程覆盖性能工程全栈——compiler optimization、memory hierarchy、parallel programming 和 performance measurement。作业要求使用 profiling-guided methodology 优化真实程序，并强制执行“先测量再优化”的纪律；正是这种纪律区分了有效性能工作和猜测式优化。
Caltech CS 179（GPU Programming with CUDA，免费讲义材料）提供结构化 GPU programming 教育。NVIDIA 的 CUDA samples 和 CUDA developer blog 提供可运行的 CUDA optimization techniques 示例及其解释。
SC（Supercomputing Conference）tutorials（免费，每年录制并发布）覆盖当前 HPC topics——programming models、performance analysis、emerging hardware——其深度符合会议的实践者受众。关于 communication-avoiding algorithms、performance portability frameworks 和 exascale programming models 的 tutorials 尤其有价值。
| 课程 | 平台 | 类型 |
|---|---|---|
| MIT 6.172 Performance Engineering（免费） | MIT OCW / YouTube | 入门 |
| Caltech CS 179 GPU Programming（免费） | Caltech course site | 入门 |
| Supercomputing（SC）tutorials（免费） | SC archive / YouTube | 参考 |
| LLNL HPC Tutorials（免费） | hpc-tutorials.llnl.gov | 入门 |
| HPC Carpentry lessons（免费） | hpc-carpentry.org | 实践 |
实践、工具与当前资料
NVIDIA Nsight（免费）是标准 GPU profiling 和 debugging 环境。Profile 一个 CUDA kernel，检查 memory access patterns，识别 warp divergence 和 occupancy limitations，是 GPU optimization 的实践工作。perf（Linux）和 Intel VTune（Intel，学生免费）提供带 hardware performance counters 的 CPU profiling，使 memory hierarchy behavior 可见。
Williams-Waterman-Patterson Roofline paper（免费，Communications of the ACM 2009）是必要的性能建模工具——短、可读，并能直接应用。把 Roofline model 应用到真实 kernels 上，也就是用 profiling tools 测量 arithmetic intensity，并将其相对 measured hardware ceilings 作图，是主要性能分析方法。
LINPACK benchmark 及其后继者（HPL、HPCG）是 supercomputer performance 的标准 metrics，可在 hpcg-benchmark.org 和 netlib.org 免费查看文档。理解 LINPACK 衡量什么（dense matrix factorization，高度规则，cache-friendly），以及它不衡量什么（真实应用性能、irregular access、memory-bandwidth dominance），是解释 Top500 rankings 的必要语境。
BLAS（Basic Linear Algebra Subprograms）和 LAPACK libraries 是 HPC 建立其上的基础数值软件。理解它们的 API 并正确使用它们——而不是重新实现基础线性代数操作——是 HPC 的基线能力。OpenBLAS、MKL 和 cuBLAS 是 CPU 和 GPU platforms 上的标准实现。
| 资源 | 平台 | 类型 |
|---|---|---|
| NVIDIA Nsight（免费） | developer.nvidia.com | 实践 |
| perf / Intel VTune（免费） | Linux / intel.com | 实践 |
| Roofline model paper（免费） | CACM 2009 | 深入 |
| BLAS / LAPACK / OpenBLAS（免费） | netlib.org / github.com/xianyi/OpenBLAS | 实践 |
| OpenMP reference guides and tutorials（免费） | openmp.org | 参考 |
| Top500 / HPCG benchmark（免费） | top500.org / hpcg-benchmark.org | 参考 |
陷阱
| 陷阱 | 为什么会误导 | 更好的回应 |
|---|---|---|
| 把 parallelism 当成 annotation | 给既有 sequential code 加上 #pragma omp parallel for 或 CUDA kernel launches，是把 parallelism 当成语法转换，而不是算法设计。结果是名义上并行、但无法扩展的代码：data dependencies 阻止有效 parallelism，synchronization overhead 吞掉 speedup，sequential bottlenecks 限制 Amdahl scaling。 |
从一开始就把 parallelism 当成算法设计关切。学习 parallel algorithms，理解哪些计算容易并行，哪些需要重新思考。写代码前，先为并行访问模式设计数据结构。选择 parallel decomposition 之前，先分析 computation 的 work 和 span。 |
| 相对于未优化 baseline 报告 speedup | 把 GPU speedup 报告为相对于未优化 sequential CPU implementation 的提升，具有误导性。使用 SIMD、cache-friendly access 和 compiler auto-vectorization 的优化 CPU implementation，可能缩小大部分表面差距。相对于朴素顺序代码声称 50× speedup，经常在相对于优化顺序代码时变成 5×。 | 始终以可获得的最佳 sequential baseline 进行测量。启用 vectorization，并优化 cache behavior 后再 profile CPU implementation；之后再判断 GPU offload 是否合理。使用 GPU acceleration 的决定，应建立在与良好优化的 sequential code 比较之上。 |
| 忽视 data movement | 关注 computational intensity 而忽视 data movement，会产生 FLOP counts 很好看、但实践性能很差的算法。多数 HPC kernels 是 bandwidth-limited，而不是 compute-limited；优化 bandwidth-limited program 的 computation 部分不会带来改进。 | 优化前应用 Roofline model。同时 profile memory bandwidth utilization 和 computation。优先考虑 cache-aware algorithm design、data layout choices 和 communication minimization。在 arithmetic intensity analysis 支持时，接受额外计算以换取更少 data movement。 |
| 把 GPU 当成 uniform speedup | GPU acceleration 经常被宣传为通用性能倍增器。但如果代码忽视 warp-level execution、产生 divergent branches、使用 non-coalesced memory access patterns，或让 device underoccupied，那么即使使用更强硬件，也可能比良好优化的 CPU code 更慢。 | 写 GPU code 之前先理解 GPU architecture。理解 warp execution、memory coalescing、shared memory 和 occupancy。使用 Nsight profile，理解时间真正花在哪里。GPU optimization 是自己的工程纪律，不能只靠学习 CUDA API 获得。 |
| 把 floating-point 当成真实算术 | 在 HPC 规模下，每次操作的 floating-point 小误差会在数百万次操作中累积。Cancellation errors、accumulated rounding 和 order-of-operations dependence 可能产生看似合理但数值错误的结果。对 scientific computing 来说，数值错误结果与错误代码同样严重，而且更难检测。 | 把 numerical analysis 与 HPC programming 一起学习。理解 floating-point error propagation、condition numbers，以及 numerically stable 和 unstable algorithms 的区别。用 known solutions 或 analytical bounds 测试数值结果。对于精度重要的应用，阅读 Higham 关于 floating-point analysis 的处理。 |
| 不 profile 就优化 | 最常见的无效 HPC optimization，是优化了错误瓶颈。工程师以为 computation 是瓶颈，但实际是 memory bandwidth；以为某个 device 上的 computation 是瓶颈，但实际是 devices 之间的 communication；以为某个 loop 很 hot，但 profiler 显示并非如此。 | 每次优化前都先 profile。使用 hardware performance counters（Nsight、perf、VTune）识别真实瓶颈。形成关于瓶颈的假设，用测量测试假设，然后再优化。先测量再优化的纪律，会让优化工作的有效性相比 intuition-based optimization 提高一个数量级。 |