English
High-performance computing studies how to solve demanding computational problems within limits on time, memory, energy, and cost. The work may use one carefully optimized node or a large cluster. The goal is a useful result at the required accuracy, not simply keeping the largest number of processors busy.
Begin with three questions: which parts of the computation can proceed independently, where the data must move, and how the numerical method affects accuracy and work. A simulation divided across machines needs both local computation and communication at partition boundaries. More machines can reduce local work while increasing coordination and data-transfer costs.
This section connects architecture (§4.1) to linear algebra (§2.3), calculus (§2.4), and scientific computing (§6.5). Parallel algorithms, performance measurement, and numerical analysis explain how a mathematical method becomes an effective implementation.
Prerequisites: Computer organization and architecture (§4.1), linear algebra (§2.3), and basic floating-point arithmetic. Numerical analysis becomes increasingly important when evaluating scientific results.
From Ballistic Tables to Exascale: The Architecture of Scale
Large numerical projects existed before electronic computers. Teams of human computers divided calculations, followed procedures, and checked results. ENIAC was built to accelerate work including ballistic-table calculation; Haigh and Ceruzzi’s account also emphasizes the people who organized and programmed such work. Faster arithmetic changed what was feasible, but useful computation still depended on decomposing the problem, supplying data, and validating the result.
The architecture of parallel computing developed through successive hardware generations, each with its own programming model and its own characteristic bottlenecks. Michael Flynn’s 1966 taxonomy — SISD, SIMD, MISD, MIMD — classified the space of possible parallel computer architectures. SIMD (Single Instruction Multiple Data) machines apply the same instruction to multiple data elements simultaneously, which fits vector processing and graphics. MIMD (Multiple Instruction Multiple Data) machines allow independent instruction streams on multiple processors, which fits general parallel computation. This classification, though simple, organized the diversity of parallel hardware and is still used in reasoning about modern processors, which combine SIMD vector units with MIMD multi-core execution.
The Cray-1, first delivered in 1976, used vector registers and pipelined functional units to process sequences of elements efficiently. A vector register could hold 64 elements, but a vector instruction did not perform all 64 element operations simultaneously in one cycle. After pipeline startup, successive elements could move through the unit at a high rate. Regular operations and an adequate supply of data were central to obtaining that throughput.
Parallel machines explored both arrays of simple processing elements and systems with independently executing processors. They did not all use the same commodity components or programming model. MPI, standardized in 1994, provided a common interface for communication among processes, including point-to-point messages and collective operations. Portable interfaces reduced dependence on vendor-specific libraries, while performance still depended on decomposition, implementation, and interconnect.
The Beowulf project at NASA Goddard in 1994 demonstrated useful parallel scientific computing with commodity computers and networking. Clusters made some workloads affordable to more research groups. Commodity parts did not remove communication costs, software effort, or operating costs; the benefit depended on matching the application to the available cluster.
Graphics processors supplied increasing arithmetic throughput and memory bandwidth for suitable parallel work. Early general-purpose GPU computation adapted graphics interfaces; CUDA, released in 2007, exposed a more direct programming model. Scientific applications could use GPUs when their kernels supplied enough parallel work and suitable data access. Host-device transfer, irregular computation, and limited device memory remained engineering constraints.
Roadrunner in 2008 and Frontier in 2022 marked petaflop and exaflop milestones on the High Performance LINPACK benchmark. An exaflop is 10^18 floating-point operations per second. A benchmark result applies to its workload and precision; it is not the speed of every scientific application. Large systems also need effective interconnects, storage, scheduling, power, cooling, and recovery to turn arithmetic capacity into usable results.
Machine-learning training shares many HPC concerns: partitioning work, exchanging gradients or activations, fitting state into memory, and choosing numerical precision. Data, tensor, and pipeline parallelism divide different parts of that work. Skills transfer between scientific and ML workloads, but their communication patterns, accuracy criteria, and failure-recovery requirements still need separate analysis. Accelerator product names are less useful as a foundation than these recurring constraints.
Parallelism, Data Movement, and Hardware at Scale
Use a model to identify a possible limit, then measure whether the implementation approaches it. Separate fixed-workload speedup, larger-problem capacity, communication costs, and numerical accuracy.
Extracting and Managing Parallelism
Parallel decomposition assigns work while respecting dependencies. Amdahl’s law models a fixed job whose serial fraction is s: ideal speedup on p processors is 1 / (s + (1-s)/p). With s = 0.1 and eight processors, speedup is about 4.71; with infinitely many, it is at most ten. Communication and imbalance can make actual speedup lower. The serial fraction and ideal division are assumptions of the model, not quantities fixed for every input size.
Strong scaling keeps the problem fixed while adding processors. Weak scaling grows the problem with processor count, commonly keeping work per processor approximately fixed. Gustafson’s scaled-workload perspective asks how much more work can be completed in a similar time. It addresses a different question from accelerating one fixed job and does not repeal Amdahl’s limit.
In the work-span model, work W is total operation cost and span S is the longest dependency chain. With p ideal processors, execution time is at least max(W/p, S), and available parallelism is bounded by W/S. Reducing span may require additional work or storage. For example, a tree reduction has logarithmic dependency depth, unlike a sequential accumulator’s linear chain, while still performing a linear number of additions.
OpenMP commonly expresses shared-memory parallel work; MPI supplies process communication, collectives, and other facilities; GPU models expose groups of many device threads. Separate MPI address spaces remove some shared-variable hazards, but do not eliminate races in threaded code, communication buffers, or shared-memory and remote-memory operations. Matching messages, completing nonblocking operations before reusing buffers, and agreeing on collective participation are correctness requirements.
A two-dimensional grid simulation illustrates the scaling cost. If each square partition has side length n, its local work can grow like n², while exchanging a one-cell boundary grows like n. Dividing a fixed grid into more, smaller partitions increases boundary traffic relative to local work. Overlapping communication with independent interior computation can help, but only when dependencies and the implementation permit it.
Data Movement and Performance Limits
Data movement can limit performance through latency, bandwidth, or energy cost. A chain of dependent loads may be latency-limited; many independent transfers may saturate bandwidth. Comparing the latency of one arithmetic instruction with one memory access does not by itself predict the throughput of a whole kernel, because operations can overlap.
The Roofline model connects arithmetic intensity I—operations per byte moved across a specified memory boundary—to bandwidth B and arithmetic capacity F. Its basic ceiling is performance ≤ min(F, B×I). For an illustrative device with F = 100 GFLOP/s, B = 50 GB/s, and I = 0.25 FLOP/byte, the bandwidth ceiling is 12.5 GFLOP/s. Increasing arithmetic capacity alone does not raise that ceiling. Actual performance may be lower because of latency, dependencies, instruction mix, or poor utilization.
Specify the precision and memory level when using this model. Traffic between registers and cache is different from traffic between device memory and a host. Raising arithmetic intensity by eliminating unnecessary transfers can help; merely adding useless arithmetic can raise the ratio without reducing time to solution. Compare useful elapsed time as well as FLOP/s.
Matrix tiling reuses blocks in fast storage before replacing them. The arithmetic can remain the same while transfers to slower storage decrease. Distributed algorithms such as Cannon’s and SUMMA organize matrix partitions and communication; some replication-based algorithms spend extra memory to reduce communication. The tradeoff is among computation, movement, capacity, and synchronization, not a universal rule that every faster method performs more arithmetic.
Floating-Point and Numerical Correctness
Floating-point arithmetic approximates real arithmetic with finite precision and range. Under ordinary binary64 rounding, (1e16 + (-1e16)) + 1 gives 1, while 1e16 + ((-1e16) + 1) gives 0. The inner addition in the second expression rounds away the unit difference. Changing a parallel reduction’s order can therefore change its result without a race or a compiler bug.
Conditioning describes the problem: how much can the exact answer change when the input changes slightly? Stability describes the algorithm: how do its rounding errors affect the answer? A backward-stable method returns the exact answer to a nearby input problem; an ill-conditioned problem can still turn that small input perturbation into a large forward error. Pivoting concerns the numerical behavior of elimination, not an improvement to the original problem’s condition number.
A deterministic reduction order can improve reproducibility, while pairwise or compensated summation can improve accuracy under appropriate assumptions. These are different goals: a reproducible answer can still be inaccurate, and compensated summation alone does not guarantee identical bits across every order or platform. State whether the requirement is a numerical tolerance, stable convergence, or bit-for-bit identity.
Mixed precision uses different representations for different stages. Lower precision can reduce storage and increase throughput; higher-precision accumulation or iterative refinement can recover accuracy for suitable problems. Validate the method’s convergence and error, not just whether a benchmark runs faster. Compare against known solutions, residuals, refinement studies, or an independently computed reference as appropriate to the problem.
What Studying This Changes
Measure time to a validated result. First establish the input, algorithm, precision, and baseline; then compare processor counts and report both speedup and resource cost. A faster kernel may leave total runtime unchanged if communication, loading, or checkpointing dominates.
A useful exercise is to compare a sequential sum with a parallel tree reduction, then vary the input magnitudes. Observe dependency depth, elapsed time, and numerical differences separately. For a grid simulation, keep the grid fixed while adding workers, then enlarge the grid with the worker count. The two experiments make strong and weak scaling concrete.
These habits transfer beyond supercomputers. Data layout, bounded dependencies, appropriate precision, and a clear performance hypothesis also improve ordinary numerical programs. Use added hardware only when the computation can make useful use of it.
Resources
Books and Texts
Pacheco and Malensek introduce parallel programming; Hager and Wellein develop performance engineering. Programming Massively Parallel Processors develops GPU programming, alongside the applicable CUDA documentation. Trefethen and Bau, Demmel, and Higham supply numerical analysis at different depths. Using MPI and the MPI specification explain communication semantics; JaJa develops parallel algorithms. Keep programming-model correctness, hardware cost, and numerical accuracy visible while reading.
| Book | Role | Type |
|---|---|---|
| Pacheco & Malensek, An Introduction to Parallel Programming (2nd ed.) | Parallel programming models and examples | Entry |
| Hager & Wellein, Introduction to HPC for Scientists and Engineers | HPC performance engineering entry | Entry |
| Hwu, Kirk & Hajj, Programming Massively Parallel Processors (4th ed.) | GPU programming and optimization | Entry |
| CUDA Programming Guide (free, NVIDIA) | Authoritative GPU programming reference | Reference |
| Trefethen & Bau, Numerical Linear Algebra | Numerical linear algebra foundation | Reference |
| Demmel, Applied Numerical Linear Algebra | Numerical LA at implementation depth | Depth |
| Higham, Accuracy and Stability of Numerical Algorithms (2nd ed.) | Floating-point error analysis | Reference |
| Gropp, Lusk & Skjellum, Using MPI (3rd ed.) | MPI programming reference | Reference |
| JaJa, An Introduction to Parallel Algorithms | Parallel algorithms foundations | Depth |
| Hennessy & Patterson, Computer Architecture: Quantitative Approach (see §4.1) | Architecture reference for HPC | Reference |
Courses and Lectures
MIT 6.172 connects optimization to measurement, and Caltech CS 179 develops GPU programming. LLNL tutorials and HPC Carpentry offer exercises in cluster use and parallel concepts. SC tutorials provide topic-specific depth; access to slides or recordings varies by offering. Use a course exercise to compare strong and weak scaling with the same explicitly described algorithm.
| Course | Platform | Type |
|---|---|---|
| MIT 6.172 Performance Engineering (free) | MIT OCW / YouTube | Entry |
| Caltech CS 179 GPU Programming (free) | Caltech course site | Entry |
| Supercomputing (SC) tutorials (access varies) | SC archive / YouTube | Reference |
| LLNL HPC Tutorials (free) | hpc-tutorials.llnl.gov | Entry |
| HPC Carpentry lessons (free) | hpc-carpentry.org | Practice |
Practice, Tools, and Current Sources
Use Nsight, perf, or VTune according to the target platform, and the Williams–Waterman–Patterson Roofline paper to relate measurements to a model. HPL measures a dense linear-algebra workload; HPCG is a complementary benchmark stressing other patterns, not simply a successor that measures all applications. BLAS specifies basic linear-algebra operations and LAPACK builds numerical routines on them; OpenBLAS, oneMKL, and cuBLAS provide overlapping but different interfaces and implementations. Check the precision, threading, and device behavior of the library actually used.
| Resource | Platform | Type |
|---|---|---|
| NVIDIA Nsight (free) | developer.nvidia.com | Practice |
| perf / Intel VTune (free) | Linux / intel.com | Practice |
| Roofline model paper (free) | CACM 2009 | Depth |
| BLAS / LAPACK / OpenBLAS (free) | netlib.org / github.com/xianyi/OpenBLAS | Practice |
| OpenMP reference guides and tutorials (free) | openmp.org | Reference |
| Top500 / HPCG benchmark (free) | top500.org / hpcg-benchmark.org | Reference |
Traps
| Trap | Why it misleads | Better response |
|---|---|---|
| Assuming a parallel annotation removes dependencies | Correctness and useful concurrency depend on the algorithm and data. | Analyze dependencies, work, span, and synchronization before comparing speed. |
| Reporting speedup against an unexplained baseline | An unoptimized baseline or omitted transfer cost can exaggerate a gain. | Describe the baseline and compare full runtimes at the same required accuracy. |
| Assuming arithmetic throughput is the only limit | Latency, bandwidth, synchronization, and imbalance can dominate. | Use a relevant model and measurements to identify the limiting resource. |
| Assuming GPU offload always helps | Launch, transfer, divergence, and insufficient work can consume the benefit. | Measure a representative kernel and its surrounding data path on the target hardware. |
| Equating reproducibility with accuracy | The same rounded answer can be consistently wrong for the required tolerance. | Check conditioning, algorithm stability, and an independent accuracy criterion. |
| Optimizing without a testable explanation | A faster component may not reduce the time that matters. | State a bottleneck hypothesis, measure, change one relevant factor, and compare validated results. |
中文
高性能计算研究怎样在时间、内存、能量与成本限制内，解决计算需求较高的问题。工作可以使用一台仔细优化的节点，也可以使用大型集群。目标是在要求的精度下得到有用结果，而不只是让尽可能多的处理器保持忙碌。
可以从三个问题开始：哪些计算部分可以独立推进，数据必须移动到哪里，以及数值方法怎样影响精度与工作量。把模拟分到多台机器后，既需要本地计算，也需要分区边界上的通信。增加机器可以减少本地工作，同时增加协调与数据传输成本。
本节把体系结构（§4.1）连接到线性代数（§2.3）、微积分（§2.4）与科学计算（§6.5）。并行算法、性能测量与数值分析，共同解释数学方法怎样成为有效实现。
前置知识：计算机组成与体系结构（§4.1）、线性代数（§2.3）与基本浮点算术。评估科学结果时，数值分析会变得越来越重要。
从弹道表到 Exascale：规模的架构
大型数值项目早于电子计算机。人工计算团队分配计算、遵循程序并核对结果。ENIAC 的建造用于加速包括弹道表计算在内的工作；Haigh 与 Ceruzzi 的叙述也强调组织和编程这些工作的人。更快的算术改变了可行范围，但有用的计算仍依赖问题分解、数据供给与结果验证。
并行计算架构经过连续几代硬件发展，每一代都有自己的 programming model 和典型瓶颈。Michael Flynn 1966 年的分类法——SISD、SIMD、MISD、MIMD——对可能的并行计算机架构空间进行了分类。SIMD（Single Instruction Multiple Data）machines 同时把同一条指令应用到多个数据元素上，适合 vector processing 和 graphics。MIMD（Multiple Instruction Multiple Data）machines 允许多个 processors 上的独立指令流，适合通用并行计算。这个分类虽然简单，却组织了并行硬件的多样性；它至今仍用于推理现代处理器，因为现代处理器结合了 SIMD vector units 和 MIMD multi-core execution。
1976 年首次交付的 Cray-1，使用向量寄存器与流水线功能单元高效处理元素序列。一个向量寄存器可容纳 64 个元素，但向量指令并非在一个周期同时完成全部 64 个元素操作。流水线启动后，后续元素可以高速连续通过功能单元。规则的操作与足够的数据供给，是获得这种吞吐量的关键。
并行机器既探索了简单处理单元阵列，也探索了处理器独立执行的系统。它们并非都采用相同通用组件或编程模型。1994 年标准化的 MPI，为进程间通信提供公共接口，包括点对点消息与集合操作。可移植接口减少了对厂商专用库的依赖，但性能仍取决于分解方式、实现与互连。
1994 年 NASA Goddard 的 Beowulf 项目，展示了使用通用计算机与网络完成有用的并行科学计算。集群使更多研究团队负担得起部分工作负载。通用组件没有消除通信、软件开发或运行成本；收益取决于应用是否适合可用集群。
图形处理器为适合的并行工作提供不断增加的算术吞吐量与内存带宽。早期 GPU 通用计算借用图形接口，2007 年发布的 CUDA 提供更直接的编程模型。当计算核具有足够并行工作与合适访存方式时，科学应用可以利用 GPU。主机与设备之间的传输、不规则计算与有限设备内存，仍然是工程约束。
2008 年的 Roadrunner 与 2022 年的 Frontier，在 High Performance LINPACK 基准上分别达到每秒千万亿次与百亿亿次浮点运算的里程碑。一个 exaflop 表示每秒 10^18 次浮点运算。基准结果适用于相应工作负载与精度，并非每个科学应用的速度。大型系统还需要有效的互连、存储、调度、供电、散热与恢复，才能把算术能力转化为有用结果。
机器学习训练与 HPC 共享许多问题：划分工作、交换梯度或激活值、把状态装入内存，以及选择数值精度。数据并行、张量并行与流水线并行划分不同部分的工作。技能可以在科学与机器学习工作负载之间迁移，但通信模式、精度标准与故障恢复要求仍需分别分析。相比加速器产品名称，这些反复出现的约束更适合作为理解基础。
Parallelism、Data Movement 与 Scale 下的硬件
先用模型找出可能的限制，再测量实现是否接近该限制。应分开考虑固定工作负载的加速、更大问题的处理能力、通信成本与数值精度。
提取并管理 Parallelism
并行分解分配工作，同时尊重依赖。Amdahl 定律描述固定任务，设串行比例为 s，在 p 个处理器上的理想加速比为 1 / (s + (1-s)/p)。当 s = 0.1、处理器为八个时，加速比约为 4.71；即使处理器无限多，也至多为十。通信与负载不均衡会使实际加速更低。串行比例与理想分配是模型假设，并非对每种输入规模都固定不变。
强扩展保持问题不变并增加处理器，弱扩展则随处理器数量扩大问题，通常使每个处理器的工作量近似不变。Gustafson 的扩展工作负载观点，询问在类似时间内可以完成多少更多工作。它与加速同一个固定任务回答不同问题，不会推翻 Amdahl 的限制。
在工作量—跨度模型中，工作量 W 是总操作成本，跨度 S 是最长依赖链。使用 p 个理想处理器时，执行时间至少为 max(W/p, S)，可用并行度受 W/S 限制。减小跨度可能需要额外工作或存储。例如，树形归约的依赖深度为对数级，而顺序累加器具有线性依赖链；两者的加法次数仍都为线性级。
OpenMP 常用于表达共享内存并行工作，MPI 提供进程通信、集合操作等设施，GPU 模型则显式提供大量设备线程的分组。独立的 MPI 地址空间消除了部分共享变量风险，但不消除线程代码、通信缓冲区，以及共享内存与远程内存操作中的竞态。匹配消息、在复用缓冲区之前完成非阻塞操作，以及一致参与集合操作，都是正确性要求。
二维网格模拟可以说明扩展成本。如果每个正方形分区边长为 n，本地工作量可能按 n² 增长，而交换一格宽边界的数据量按 n 增长。把固定网格切成更多、更小的分区，会提高边界通信相对本地工作的比例。把通信与独立的内部计算重叠可能有帮助，但需要依赖关系与实现允许。
数据移动与性能限制
数据移动可以通过延迟、带宽或能耗限制性能。相互依赖的加载链可能受延迟限制，大量独立传输则可能耗尽带宽。仅比较一条算术指令与一次内存访问的延迟，无法直接预测整个计算核的吞吐量，因为操作可以重叠。
Roofline 模型把算术强度 I——跨指定存储边界传输每字节所对应的操作数——与带宽 B、算术能力 F 连接起来。基本上界为 性能 ≤ min(F, B×I)。假设某设备 F = 100 GFLOP/s、B = 50 GB/s、I = 0.25 FLOP/byte，带宽上界就是 12.5 GFLOP/s。只增加算术能力不能提高这个上界。实际性能还可能因延迟、依赖、指令组合或利用率不足而更低。
使用模型时，要说明精度与存储层次。寄存器与缓存之间的流量，不同于设备内存与主机之间的流量。消除不必要传输以提高算术强度可以有帮助；仅增加无用算术，也能提高比值，却未必缩短求解时间。除了 FLOP/s，还要比较完成有用工作的耗时。
矩阵分块在替换数据之前，重复使用快速存储中的块。算术可以保持不变，而访问较慢存储的传输减少。Cannon、SUMMA 等分布式算法组织矩阵分区与通信，某些基于复制的算法则花费额外内存以减少通信。这里权衡的是计算、移动、容量与同步，并非所有更快方法都必须增加算术操作。
Floating-Point 与数值正确性
浮点算术用有限精度与范围近似实数算术。在通常的 binary64 舍入下，(1e16 + (-1e16)) + 1 得到 1，而 1e16 + ((-1e16) + 1) 得到 0。第二个表达式的内层加法，把一个单位的差异舍入掉了。因此，改变并行归约顺序，即使没有竞态或编译器错误，也可以改变结果。
条件性描述问题：输入略微变化时，精确答案可能变化多少？稳定性描述算法：舍入误差怎样影响答案？后向稳定方法返回某个邻近输入问题的精确答案；病态问题仍可能把很小的输入扰动放大为很大的前向误差。选主元关系到消元的数值行为，并不改善原问题的条件数。
确定性的归约顺序可以改善可复现性，成对求和或补偿求和则可以在适当假设下改善精度。这是不同目标：可复现答案仍可能不准确，补偿求和本身也不保证在每种顺序或平台下逐位相同。需要说明要求究竟是数值容差、稳定收敛，还是逐位一致。
混合精度在不同阶段使用不同表示。较低精度可以减少存储并提高吞吐量；对于合适的问题，较高精度累加或迭代改进可以恢复精度。需要验证方法的收敛与误差，而不只是基准是否更快。应按问题特点，比较已知解、残差、逐步细化实验或独立计算的参考结果。
学习这一部分会改变什么
测量得到经过验证结果所需的时间。先确定输入、算法、精度与基线，再比较处理器数量，同时报告加速比与资源成本。如果通信、加载或检查点占主要时间，更快的计算核可能不改变总运行时间。
一个有用练习是比较顺序求和与并行树形归约，再改变输入数值的量级，分别观察依赖深度、运行时间与数值差异。对于网格模拟，可以先固定网格增加工作进程，再随工作进程数量扩大网格。两种实验使强扩展与弱扩展变得具体。
这些习惯也适用于超级计算机之外的程序。数据布局、受控的依赖、合适精度与明确性能假设，同样改善普通数值程序。只有计算能够有效利用时，新增硬件才有意义。
资源
书籍与文本
Pacheco 与 Malensek 介绍并行编程，Hager 与 Wellein 展开性能工程。Programming Massively Parallel Processors 配合适用的 CUDA 文档，讲述 GPU 编程。Trefethen 与 Bau、Demmel，以及 Higham 提供不同深度的数值分析。Using MPI 与 MPI 规范解释通信语义，JaJa 展开并行算法。阅读时应同时关注编程模型正确性、硬件成本与数值精度。
| 书籍 | 作用 | 类型 |
|---|---|---|
| Pacheco & Malensek, An Introduction to Parallel Programming（第 2 版） | 并行编程模型与实例 | 入门 |
| Hager & Wellein, Introduction to HPC for Scientists and Engineers | HPC performance engineering 入口 | 入门 |
| Hwu、Kirk 与 Hajj，Programming Massively Parallel Processors（第 4 版） | GPU 编程与优化 | 入门 |
| CUDA Programming Guide（免费，NVIDIA） | 权威 GPU programming 参考 | 参考 |
| Trefethen & Bau, Numerical Linear Algebra | Numerical linear algebra 基础 | 参考 |
| Demmel, Applied Numerical Linear Algebra | 实现深度的 numerical LA | 深入 |
| Higham, Accuracy and Stability of Numerical Algorithms（第 2 版） | Floating-point error analysis | 参考 |
| Gropp, Lusk & Skjellum, Using MPI（第 3 版） | MPI 编程参考 | 参考 |
| JaJa, An Introduction to Parallel Algorithms | Parallel algorithms 基础 | 深入 |
| Hennessy & Patterson, Computer Architecture: Quantitative Approach（见 §4.1） | HPC 所需体系结构参考 | 参考 |
课程与讲座
MIT 6.172 把优化连接到测量，Caltech CS 179 展开 GPU 编程。LLNL 教程与 HPC Carpentry 提供集群使用和并行概念练习，SC 教程提供具体主题的深度；幻灯片和录像的获取依场次而异。可以用课程练习，在同一个明确描述的算法上比较强扩展与弱扩展。
| 课程 | 平台 | 类型 |
|---|---|---|
| MIT 6.172 Performance Engineering（免费） | MIT OCW / YouTube | 入门 |
| Caltech CS 179 GPU Programming（免费） | Caltech course site | 入门 |
| Supercomputing（SC）教程（获取方式不一） | SC archive / YouTube | 参考 |
| LLNL HPC Tutorials（免费） | hpc-tutorials.llnl.gov | 入门 |
| HPC Carpentry lessons（免费） | hpc-carpentry.org | 实践 |
实践、工具与当前资料
按目标平台使用 Nsight、perf 或 VTune，并用 Williams、Waterman 与 Patterson 的 Roofline 论文 把测量连接到模型。HPL 测量稠密线性代数工作负载，HPCG 是强调其他模式的互补基准，并非能测量所有应用的简单后继。BLAS 规定基础线性代数操作，LAPACK 在其上构造数值例程；OpenBLAS、oneMKL 与 cuBLAS 提供相互重叠但不同的接口和实现。应检查实际所用库的精度、线程与设备行为。
| 资源 | 平台 | 类型 |
|---|---|---|
| NVIDIA Nsight（免费） | developer.nvidia.com | 实践 |
| perf / Intel VTune（免费） | Linux / intel.com | 实践 |
| Roofline model paper（免费） | CACM 2009 | 深入 |
| BLAS / LAPACK / OpenBLAS（免费） | netlib.org / github.com/xianyi/OpenBLAS | 实践 |
| OpenMP reference guides and tutorials（免费） | openmp.org | 参考 |
| Top500 / HPCG benchmark（免费） | top500.org / hpcg-benchmark.org | 参考 |
陷阱
| 误区 | 为什么会误导 | 更好的做法 |
|---|---|---|
| 假设并行标记消除依赖 | 正确性与有效并发依赖算法和数据。 | 比较速度前，分析依赖、工作量、跨度与同步。 |
| 相对未解释的基线报告加速 | 未优化基线或遗漏传输成本，可能夸大收益。 | 说明基线，在相同精度要求下比较完整耗时。 |
| 认为算术吞吐量是唯一限制 | 延迟、带宽、同步与负载不均衡可以占主要部分。 | 使用相关模型与测量，识别限制资源。 |
| 假设转移到 GPU 总有帮助 | 启动、传输、分歧与工作不足可能耗尽收益。 | 在目标硬件上测量代表性计算核及其前后数据路径。 |
| 把可复现等同于准确 | 相同的舍入答案，也可能始终不满足要求的容差。 | 检查条件性、算法稳定性与独立精度标准。 |
| 没有可检验解释就优化 | 更快的组件未必减少真正关心的耗时。 | 说明瓶颈假设，测量，改变一个相关因素，再比较经过验证的结果。 |