English

Computer graphics is the discipline of generating visual content through computation: determining what color each pixel on a screen should be, given a description of geometry, materials, lights, and viewpoint. This sounds mechanical and turns out to be deep. The physics of how light interacts with matter — how it reflects off surfaces, scatters through translucent volumes, refracts through glass, bounces many times between objects before reaching the eye — produces an integral over an infinite-dimensional space that no computer can evaluate exactly. Every rendering system is therefore an approximation, and the art is in choosing which approximations to make, for which purposes, at what computational cost.

Visual computing extends this in both directions. Upstream: how is the geometry, material, and animation data represented in the first place? Meshes, parametric surfaces, implicit functions, and neural representations each have different tradeoffs for what operations they support and at what cost. Downstream: how is the rendered image understood, modified, or used to generate new images? Computer vision, image processing, and the recent generation of neural rendering and generative image models all live here. Graphics in the narrow sense — rendering — is surrounded by a broader discipline that covers the entire pipeline from scene representation through display.

This section covers the core discipline: rendering (how scenes are converted to images), real-time graphics (how this can happen thirty or sixty times per second), and visual representation (how scenes are described). The connections to deep learning (§5.3) and to scientific computing (§6.5) are real and growing — neural rendering has become a subdiscipline of its own, and simulation sits at the intersection of graphics and scientific computing — but those connections are downstream of this core.

Prerequisites: Linear algebra (§2.3) — transformations, projections, and lighting are fundamentally linear-algebraic. Calculus (§2.4) — rendering is numerical integration. Probability and statistics (§2.5) — Monte Carlo rendering is statistical sampling. Computer architecture (§4.1) — GPU programming requires understanding the hardware.

From Oscilloscopes to Neural Radiance Fields

The founding act of computer graphics is usually dated to Ivan Sutherland’s Sketchpad system, which he described in his 1963 MIT dissertation. Sketchpad allowed users to draw on a CRT display using a light pen, creating geometric shapes that could be copied, scaled, and constrained. The system introduced concepts that remain central: graphical objects with properties that could be edited, the separation of the object representation from its displayed form, and constraints as a way of specifying relationships. Sutherland received the Turing Award in 1988, described at the time as “the father of computer graphics.”

The early period was necessarily concerned with lines and wireframes, because filling polygons with realistic shading required computational resources that did not exist. The breakthrough work of the 1970s was shading. Henri Gouraud described per-vertex color interpolation in 1971 — smooth shading over a polygon from colors computed at its vertices. Bui-Tuong Phong described per-pixel normal interpolation in 1975, producing smoother specular highlights, and proposed the Phong reflection model — a simple mathematical description of how light reflects from surfaces that became the standard shading model for twenty years. Jim Blinn extended Phong’s model in 1977, and Blinn-Phong shading remained the default approach in real-time graphics until physically-based rendering displaced it in the 2010s.

Ray tracing was articulated as a rendering approach by Arthur Appel in 1968, but Turner Whitted’s 1980 SIGGRAPH paper “An Improved Illumination Model for Shaded Display” demonstrated its power for global illumination: reflections, refraction, and shadows all naturally fell out of tracing rays through the scene. Whitted’s images were striking precisely because they showed effects — reflections of reflections in a glass ball — that rasterization could not produce. Ray tracing was computationally expensive and for two decades remained a technique for offline rendering only, but it established the path-tracing approach to global illumination that would eventually become dominant.

The Monte Carlo approach to rendering — using random sampling to estimate the integral over all light paths — was formalized by Kajiya’s 1986 SIGGRAPH paper “The Rendering Equation,” which expressed all light transport as a single integral equation. This framing was the intellectual foundation for the path tracing, bidirectional path tracing, and Metropolis light transport algorithms that followed. Eric Veach’s 1997 Stanford dissertation “Robust Monte Carlo Methods for Light Transport Simulation” established the theoretical framework for modern production rendering and remains the standard reference for anyone doing serious work with Monte Carlo rendering.

The film and animation industry was the early adopter of offline rendering. Pixar, founded in 1986 as a spinoff from Lucasfilm’s Computer Division, developed RenderMan — a rendering API and shading language that became the film industry standard. Toy Story in 1995, rendered entirely in computer graphics, demonstrated what physically-modeled geometry, animation, and shading could achieve for storytelling. Every subsequent Pixar film pushed the technical state of the art, and Pixar’s SIGGRAPH papers were among the most influential in the field through the 1990s and 2000s.

The real-time graphics revolution came from the GPU. The first consumer graphics cards handled transformation and rasterization in fixed hardware; by the late 1990s, texture mapping, lighting, and basic effects were hardware-accelerated. NVIDIA’s GeForce 256 in 1999 performed the transform and lighting calculation in hardware, replacing the CPU computation that had limited real-time graphics. The introduction of programmable vertex and pixel shaders in 2001-2002 — with DirectX 8 and later shader model versions — transformed the GPU from a fixed-function pipeline into a general parallel computer for graphics. This enabled the explosion of real-time visual effects: dynamic shadows, normal mapping, ambient occlusion, complex material effects — all implemented as shader programs running on the GPU.

The physically-based rendering (PBR) movement brought the rigor of offline rendering to real-time graphics. Matt Pharr, Greg Humphreys, and Wenzel Jakob’s Physically Based Rendering: From Theory to Implementation (PBRT), first published in 2004, provided the accessible treatment that made Monte Carlo rendering understandable to the broader community. The GPU programmability of the 2000s made it possible to implement physically-grounded material models in real-time shaders. Walt Disney Animation Studios published their material model in 2012, establishing the “Disney PBR” approach — a physically plausible BRDF (Bidirectional Reflectance Distribution Function) parameterized for artistic control — which became the standard in game engines and real-time graphics through the 2010s.

NVIDIA’s RTX 20-series GPUs in 2018 introduced dedicated hardware for ray tracing acceleration (the RT cores), making real-time ray tracing practically viable for the first time. Microsoft’s DirectX Raytracing API and the Vulkan ray tracing extension provided the programming interfaces. Hybrid rendering — rasterization for the primary visibility pass, ray tracing for reflections, shadows, and ambient occlusion — became the standard approach for AAA games by the early 2020s. Lumen, Unreal Engine 5’s global illumination system, and similar systems demonstrated that production-quality real-time global illumination was achievable on consumer hardware.

The neural rendering revolution began with Ben Mildenhall and colleagues’ Neural Radiance Fields (NeRF) paper in 2020. NeRF represented a scene as a neural network: given a 3D position and viewing direction, the network predicted the density and color at that point. Rendering a novel view required integrating the network along rays from the camera — expensive, but producing photo-realistic results from a set of input photographs without explicit geometry reconstruction. The approach transformed 3D reconstruction and generated immediate follow-on work: faster variants (Instant NGP, 2022), better quality, extension to dynamic scenes. Gaussian Splatting (Kerbl et al., 2023) replaced the neural network with 3D Gaussians, achieving real-time rendering speed for neural representations — a significant development that made NeRF-like representations practical for interactive use.

The generative model revolution — diffusion models producing photorealistic images and videos from text prompts — is not primarily a graphics development but has become a graphics concern. AI-generated textures, AI-assisted asset creation, AI denoising in Monte Carlo rendering (NVIDIA’s DLSS and Intel’s XeSS use neural networks to upscale lower-resolution renders), and AI-generated images as training data for neural rendering — all represent intersections between generative models and the graphics pipeline that are reshaping workflows.

Rendering, Representation, and Real-Time Performance

The Rendering Pipeline: From Scene to Pixels

Rendering is the computation of an image from a scene description. Two families of approaches dominate: rasterization and ray tracing. They differ in how they determine the relationship between scene elements and image pixels.

Rasterization projects geometry (triangles) onto the image plane and determines which pixels each triangle covers. For each covered pixel, a fragment shader program computes the color, typically using the material properties, surface normal, and lighting. Rasterization is efficient because the projection and coverage computation are highly parallelizable and map naturally to GPU architecture. The GPU processes millions of triangles per frame across thousands of parallel shader units. The limitation is that global effects — accurate reflections, refractions, global illumination — require explicit approximations, because the rasterization pipeline sees only the geometry visible from the camera.

Ray tracing casts rays from the camera into the scene and computes intersections with geometry. For each intersection, secondary rays (reflection rays, shadow rays, refraction rays) can be cast to compute global effects. The advantage is physical accuracy: reflections, refractions, caustics, and soft shadows emerge naturally from the ray tracing calculation. The disadvantage is cost: tracing rays into arbitrary scene geometry requires efficient spatial data structures (BVH, k-d trees), and each secondary ray can spawn further rays, making the computation tree-structured and difficult to parallelize efficiently. GPU ray tracing uses dedicated hardware to accelerate the BVH traversal and ray-geometry intersection computations.

Production rendering uses path tracing: traces paths of light through the scene by sampling light paths randomly. A camera ray hits a surface; a new ray is sampled from the material’s BSDF (Bidirectional Scattering Distribution Function); that ray hits another surface; and so on until the path escapes the scene or is terminated by Russian Roulette. The integral being computed — the rendering equation — is estimated by averaging many such path samples. Path tracing is unbiased: with enough samples, it converges to the correct answer. Variance (noise in the image) decreases as 1/√n with n samples, requiring many samples for clean images.

The physically-based rendering equation, derived by Kajiya, expresses the emitted radiance L_o from a surface point x in direction ω_o as:

L_o(x, ω_o) = L_e(x, ω_o) + ∫_{Ω} f_r(x, ω_i, ω_o) L_i(x, ω_i) (ω_i · n) dω_i

where L_e is emitted radiance, f_r is the BRDF, and L_i is incoming radiance from direction ω_i. Monte Carlo path tracing estimates this integral by sampling incoming directions from the hemisphere Ω. The BRDF f_r encodes the material’s appearance — how it reflects light at different angles.

Geometric Representation

Before a scene can be rendered, it must be represented. Different representations support different operations and have different efficiency tradeoffs.

Triangle meshes are the universal format for real-time rendering: sequences of triangles sharing vertices and normals, with additional attributes (texture coordinates, tangents) for materials. The GPU pipeline is optimized for triangle mesh rasterization. Meshes are compact, easily processed in parallel, and supported by every rendering API. Their limitation is that representing smooth surfaces requires many triangles, and editing mesh topology is awkward.

Parametric surfaces (NURBS, subdivision surfaces) represent smooth curved surfaces explicitly and are used in modeling and in film rendering, where geometry is tessellated into triangles at render time. Subdivision surfaces are the standard in Pixar-style character work: a coarse control mesh is subdivided according to rules that produce a smooth limit surface, providing exact smooth geometry without requiring explicit specification.

Signed distance fields (SDFs) represent geometry as the signed distance to the surface at each point in a volume. Positive values are outside; negative values are inside; zero is on the surface. SDFs support efficient ray marching — march along a ray taking steps of size equal to the SDF value at the current point, guaranteed not to miss the surface — and are used in procedural geometry, font rendering, collision detection, and soft-body simulation. Neural signed distance fields, part of the NeRF family, represent implicit surfaces as neural networks.

Neural representations (NeRF, Gaussian Splatting) represent a scene directly as a function mapping position and view direction to appearance. NeRF uses a neural network; Gaussian Splatting uses a set of 3D Gaussians with position, scale, rotation, color, and opacity. These representations are acquired from photographs rather than hand-modeled, enabling photorealistic reconstruction of real scenes, but they are not easily edited and require novel-view synthesis rather than explicit geometric operations.

GPU Architecture and Real-Time Performance

The GPU’s advantage for graphics comes from massive parallelism: thousands of simple processors executing the same shader program on different data simultaneously (SIMT, Single Instruction Multiple Threads). A vertex shader runs on every vertex; a fragment shader runs on every covered fragment; a compute shader runs on a dispatch of work items. The efficiency depends on keeping all those processors busy with coherent work.

The memory hierarchy matters as much as the compute units. GPU memory bandwidth — typically much higher than CPU bandwidth — determines how fast the GPU can read textures and vertex data. L1 and L2 caches reduce bandwidth for coherent access patterns. Texture samplers, hardware texture caches with bilinear and trilinear interpolation, accelerate the most common memory access pattern in fragment shaders.

Performance bottlenecks in real-time rendering are classified by which stage of the pipeline is limiting throughput. Vertex-bound applications are limited by the rate of vertex processing; the fix is reducing vertex count or simplifying vertex shaders. Fragment-bound applications are limited by fragment shader execution; reducing overdraw (pixels drawn multiple times) or simplifying fragment shaders helps. Bandwidth-bound applications are limited by texture or geometry memory access; reducing texture resolution, compressing textures, or improving spatial locality helps. Identifying which bottleneck applies requires profiling with GPU-specific tools (RenderDoc, NSight, PIX).

Level of detail (LOD) systems manage scene complexity by using simplified versions of geometry and materials for objects far from the camera, where detail is not visible. Nanite (Unreal Engine 5’s virtualized geometry system) takes this to an extreme: meshes with arbitrary polygon counts are streamed and rasterized at only the screen-pixel resolution needed, eliminating the need to hand-author multiple LOD levels.

What Studying This Changes

Computer graphics changes how practitioners see images and understand what computation can create.

The first change is visual literacy about rendering. The practitioner who has studied physically-based rendering can look at a rendered image and analyze it: is this lighting consistent with the claimed light positions? Is this material physically plausible? What sampling artifact is causing this noise pattern? Is this shadow geometrically correct? This analysis capability — seeing images as the output of a physical simulation rather than as aesthetic choices — is not something non-practitioners develop and is valuable both for production work and for evaluating AI-generated images.

The second change is GPU programming intuition. After studying real-time graphics, the practitioner understands what operations are cheap and expensive on a GPU, what memory access patterns are efficient and inefficient, how to read a GPU performance profile and identify bottlenecks. This knowledge transfers beyond graphics to any GPU-accelerated computation — ML training and inference, scientific simulation, parallel data processing.

The third change is the mathematical vocabulary for spatial reasoning. Transformations, projections, coordinate systems, homogeneous coordinates, quaternions, barycentric coordinates — these mathematical tools have applications throughout computer science and engineering, but they are acquired most naturally in the graphics context where their visual meaning is concrete. The practitioner who has implemented a projection matrix understands what it is doing in a way that someone who has only applied it does not.

The fourth change is appreciation for the physical basis of visual appearance. Physically-based rendering requires understanding how light interacts with materials — absorption, scattering, refraction, diffraction. This physical understanding is useful not just for rendering but for evaluating visual claims more broadly: whether a photograph is consistent with the stated lighting conditions, whether a material in a scene is plausible, whether an AI-generated image is physically consistent.

Resources

Books and Texts

Pharr, Jakob, and Humphreys’s Physically Based Rendering: From Theory to Implementation (PBRT, 4th ed., 2023, free at pbr-book.org) is the most important single text in offline rendering. It provides both the mathematical theory — the rendering equation, Monte Carlo integration, BSDF models — and a complete reference path tracer implementation, carefully written and extensively commented. Reading PBRT and implementing portions of it is the definitive route to understanding production rendering. The fourth edition covers contemporary developments including GPU rendering and volumetric scattering. Freely available online.

Akenine-Möller, Haines, Hoffman, Pesce, Iwanicki, and Hillaire’s Real-Time Rendering (4th ed., 2018, realtimerendering.com) is the canonical reference for real-time graphics. It covers the GPU pipeline, shading models, global illumination approximations, anti-aliasing, texturing, and dozens of specific techniques, with extensive bibliographic references. Too comprehensive to read sequentially; valuable as a reference for understanding any specific technique and for understanding where techniques fit in the broader landscape.

Hughes, van Dam, McGuire, Sklar, Foley, Feiner, and Akeley’s Computer Graphics: Principles and Practice (3rd ed., 2013) is the comprehensive foundation textbook, covering the full breadth of computer graphics with mathematical depth. More academic than PBRT or RTR, and somewhat dated in its treatment of contemporary GPU-centric practice, but unmatched for developing the mathematical foundations.

For the mathematical preparation specifically: Lengyel’s Mathematics for 3D Game Programming and Computer Graphics (3rd ed., 2011) covers linear algebra, trigonometry, calculus, and the specific mathematical structures (quaternions, homogeneous coordinates, intersection algorithms) used in graphics. Less comprehensive than a standalone mathematics text, but more directly applicable to graphics work.

Gregory’s Game Engine Architecture (3rd ed., 2018) is the canonical text for game engine systems — rendering pipeline, physics, animation, audio, the input system, and how they integrate. For practitioners pursuing game development, it bridges the gap between the graphics concepts in PBRT and RTR and the engineering reality of production game engines.

Book Role Type
Pharr, Jakob & Humphreys, Physically Based Rendering (4th ed., free) Offline rendering canonical text + implementation Entry
Akenine-Möller et al., Real-Time Rendering (4th ed.) Real-time rendering canonical reference Reference
Hughes et al., Computer Graphics: Principles and Practice (3rd ed.) Comprehensive foundational textbook Reference
Lengyel, Mathematics for 3D Game Programming and Computer Graphics Graphics mathematics Entry
Gregory, Game Engine Architecture (3rd ed.) Game engine systems Depth
Botsch et al., Polygon Mesh Processing (free) Geometry processing Depth
Bridson, Fluid Simulation for Computer Graphics (2nd ed.) Physics simulation Depth
Munzner, Visualization Analysis and Design Information visualization Entry
Veach, PhD thesis (free) Monte Carlo rendering foundations Depth
The Graphics Codex (free samples; paid app/full access may apply) Compact reference for graphics mathematics, rendering, and implementation details Reference

Courses and Lectures

Stanford CS148 (Introduction to Computer Graphics and Imaging) and CS348b (Image Synthesis) have freely available materials and cover the mathematical and algorithmic foundations of rendering, shading, and ray tracing.

UC Berkeley CS184/284 (Computer Graphics) provides a strong course with freely available materials, covering rasterization, ray tracing, modeling, and animation with implementation projects.

Cem Yuksel’s YouTube lecture series (free) is the most comprehensive freely available video introduction to computer graphics, covering shading, rendering, ray tracing, and GPU programming with clear mathematical explanations.

The Book of Shaders (Patricio Gonzalez Vivo and Jen Lowe, thebookofshaders.com, free) is an interactive introduction to shader programming — fragment shaders in GLSL — with visual examples that run in the browser. The best entry to GPU programming for beginners.

LearnOpenGL (learnopengl.com, free) is the standard tutorial site for OpenGL, covering the rendering pipeline, shaders, textures, lighting, and more advanced topics. The right practical entry for implementing graphics programs.

Course Platform Type
Stanford CS148/CS348b (free) Stanford course site Entry
UC Berkeley CS184/284 (free) Berkeley course site Entry
Cem Yuksel YouTube lectures (free) YouTube Practice
The Book of Shaders (free) thebookofshaders.com Practice
LearnOpenGL (free) learnopengl.com Practice
Scratchapixel (free) scratchapixel.com Entry
WebGPU Fundamentals (free) webgpufundamentals.org Practice
Vulkan Tutorial / Khronos Vulkan Learn (free) vulkan-tutorial.com / vulkan.org Practice

Practice, Tools, and Current Sources

The PBRT source code (free at github.com/mmp/pbrt-v4) is the implementation that accompanies the PBRT textbook. Compiling and running it, then modifying specific integrators or BSDF implementations, is the most concentrated learning exercise in rendering. The code is exemplary C++ and represents how production-quality rendering infrastructure is structured.

RenderDoc (free, renderdoc.org) is the GPU debugger of choice for graphics debugging — it captures a frame, allows stepping through draw calls, and shows the state of every render target and resource at each stage. Using it on any graphics application immediately reveals the rendering pipeline in a concrete form.

Shadertoy (shadertoy.com, free) is a browser-based environment for writing fragment shaders. The community has created thousands of shaders demonstrating every graphics technique from basic ray marching to complete path tracers. Reading and modifying community shaders is the most efficient way to learn practical shader programming.

Blender (free, blender.org) is a professional 3D creation suite with a path-tracing renderer (Cycles), real-time viewport renderer (Eevee), modeling tools, and Python scripting. Its source code is available and well-organized; using it as a rendering engine while reading PBRT clarifies the connection between theory and implementation.

The Mildenhall et al. NeRF paper (2020, free at arxiv.org) and the Kerbl et al. 3D Gaussian Splatting paper (2023, free at arxiv.org) are primary sources for neural rendering. The accompanying code releases are available on GitHub. Working through the NeRF code is the entry to neural rendering.

Resource Platform Type
PBRT source code (free) github.com/mmp/pbrt-v4 Practice
RenderDoc (free) renderdoc.org Practice
Shadertoy (free) shadertoy.com Practice
Blender (free) blender.org Practice
Mildenhall et al., NeRF code (free) GitHub Practice
Kerbl et al., Gaussian Splatting code (free) GitHub Practice
SIGGRAPH proceedings (free after delay) dl.acm.org Reference
Inigo Quilez’s website (free) iquilezles.org Practice

Traps

Trap Why it misleads Better response
Treating graphics as API calls Graphics APIs (OpenGL, Vulkan, DirectX, game engine APIs) provide access to the GPU pipeline, but the API is not the subject — the rendering algorithms and mathematical foundations are. Practitioners who learn APIs without the underlying mathematics cannot reason about what the API is doing, diagnose rendering artifacts at their source, or adapt techniques to novel problems. Study the mathematics and algorithms before the APIs. Implement a simple ray tracer in software before implementing a rasterizer with OpenGL. Understanding what the GPU is computing makes the API comprehensible; understanding the API alone leaves the computation opaque.
Skipping the mathematics Graphics depends on linear algebra (transformations, projections), calculus (integration for rendering), and probability (Monte Carlo sampling). Learners who avoid the mathematics reach a ceiling quickly — they can implement tutorials but cannot extend them. The PBRT book derives everything from first principles; spending time with the derivations rather than skipping to the code is the investment that produces durable competence. Work through the mathematical foundations before the rendering algorithms. The linear algebra prerequisites in §2.3, the calculus in §2.4, and the probability in §2.5 are all directly load-bearing. The PBRT book’s first three chapters are an appropriate starting point for understanding what mathematics rendering requires.
Offline rendering vs. real-time rendering as separate subjects Many learners specialize in either real-time (GPU, game engine, 60fps) or offline rendering (path tracing, film, hours per frame) and treat them as unrelated. In fact they share mathematical foundations (the rendering equation, BRDFs, light transport), and modern real-time rendering is increasingly using ray tracing techniques. Study both traditions. Read PBRT for offline rendering foundations, RTR for real-time rendering. The contemporary hybrid rendering pipelines that combine rasterization and ray tracing are only legible to practitioners who understand both.
Neural rendering as a replacement for classical rendering Neural rendering methods (NeRF, Gaussian Splatting) have remarkable capabilities for novel-view synthesis from photographs but are not general replacements for classical rendering: they do not support dynamic scenes easily, do not support explicit editing of geometry, and require the input photographs in the first place. Practitioners who abandon classical rendering for neural approaches discover these limitations when trying to use them in production. Study classical rendering first, then neural rendering as an extension. Understanding what path tracing is computing makes NeRF’s alternative — representing the scene as a neural function that path-tracing interprets — legible. Without the classical foundation, neural rendering is a black box.
Avoiding GPU debugging tools Graphics bugs are visual — wrong colors, missing geometry, incorrect shading, corrupted textures. Diagnosing them requires seeing what the GPU is computing at each stage, which requires GPU debugging tools (RenderDoc, NSight). Practitioners who debug graphics programs by print statements and guessing take ten times as long as practitioners who capture a frame in RenderDoc, inspect each render target, and find the exact stage where the error appears. Learn RenderDoc at the beginning of GPU programming, not after months of struggling without it. Capture a frame of any graphics application — even a tutorial — and walk through the draw calls. The GPU pipeline becomes concrete and legible in a way that reading about it cannot provide.
Treating visual output as binary correct/incorrect Rendered images are not binary — they have degrees of quality, visible artifacts with specific causes, and tradeoffs between different quality dimensions. Practitioners who treat rendering as “it looks fine” versus “it’s broken” cannot improve quality systematically. Develop visual literacy by analyzing rendered images critically: what is the noise pattern, what causes it, how would a different sampler change it? What are the aliasing artifacts? Is the ambient occlusion physically plausible? Reading the chapter on artifacts in RTR and then looking for those artifacts in rendered images develops the critical eye that production graphics requires.

中文

计算机图形学是一门通过计算生成视觉内容的学科:给定几何体、材质、光源和视角的描述,判断屏幕上每个像素应该是什么颜色。这听起来很机械,但实际非常深。光与物质如何相互作用——如何从表面反射,如何在半透明体积中散射,如何穿过玻璃发生折射,如何在到达眼睛之前在物体之间反复弹射——会产生一个定义在无限维空间上的积分,而没有任何计算机能够精确计算它。因此,每一个渲染系统都是近似;真正的艺术在于,为了什么目的、在什么计算成本下,选择哪些近似。

视觉计算则向两个方向扩展了这一问题。上游:几何、材质和动画数据最初如何被表示?网格、参数曲面、隐式函数和神经表示,各自支持不同操作,也有不同成本。下游:渲染出的图像如何被理解、修改,或者用于生成新图像?计算机视觉、图像处理,以及近年出现的神经渲染和生成式图像模型,都位于这里。狭义的图形学——渲染——被一个更大的学科包围;这个学科覆盖从场景表示到显示的整个流水线。

本节覆盖核心学科:渲染(场景如何被转换为图像)、实时图形学(如何每秒完成三十次或六十次渲染),以及视觉表示(场景如何被描述)。它与深度学习(§5.3)和科学计算(§6.5)的连接真实且不断增长——神经渲染已经成为一个独立子学科,仿真也位于图形学和科学计算的交叉处——但这些连接都发生在这个核心之后。

前置知识:线性代数(§2.3)——变换、投影和光照本质上都是线性代数问题。微积分(§2.4)——渲染是数值积分。概率与统计(§2.5)——蒙特卡洛渲染是统计采样。计算机体系结构(§4.1)——GPU 编程要求理解硬件。

从示波器到神经辐射场

计算机图形学的奠基事件通常被追溯到 Ivan Sutherland 的 Sketchpad 系统,他在 1963 年的 MIT 博士论文中描述了该系统。Sketchpad 允许用户用光笔在 CRT 显示器上绘图,创建可以复制、缩放和约束的几何形状。这个系统引入了至今仍然核心的概念:具有可编辑属性的图形对象、对象表示与显示形式的分离,以及用约束来指定关系。Sutherland 于 1988 年获得图灵奖,当时被称为“计算机图形学之父”。

早期阶段必然关注线条和线框,因为用真实明暗填充多边形所需的计算资源还不存在。1970 年代的突破性工作是着色。Henri Gouraud 于 1971 年描述了逐顶点颜色插值——从多边形顶点计算出的颜色,在多边形表面上进行平滑着色。Bui-Tuong Phong 于 1975 年描述了逐像素法线插值,产生更平滑的高光,并提出了 Phong 反射模型——一种简单的数学模型,用来描述光如何从表面反射,后来成为二十年的标准着色模型。Jim Blinn 于 1977 年扩展了 Phong 模型,而 Blinn-Phong 着色一直是实时图形学中的默认方法,直到 2010 年代被基于物理的渲染取代。

Arthur Appel 于 1968 年把光线追踪表述为一种渲染方法,但 Turner Whitted 在 1980 年 SIGGRAPH 论文 “An Improved Illumination Model for Shaded Display” 中展示了它在全局光照中的力量:反射、折射和阴影都会自然地从穿过场景的光线追踪中产生。Whitted 的图像之所以令人震撼,正是因为它展示了光栅化无法产生的效果——例如玻璃球中反射的反射。光线追踪计算成本很高,之后二十年里仍然主要是离线渲染技术,但它确立了通向全局光照的路径追踪路线,而这一路线最终会成为主导。

渲染中的蒙特卡洛方法——用随机采样估计所有光线路径上的积分——由 Kajiya 1986 年 SIGGRAPH 论文 “The Rendering Equation” 形式化,该论文把所有光传输表达为一个单一积分方程。这个框架成为后来路径追踪、双向路径追踪和 Metropolis 光传输算法的思想基础。Eric Veach 1997 年的斯坦福博士论文 “Robust Monte Carlo Methods for Light Transport Simulation” 建立了现代生产渲染的理论框架,至今仍是任何认真从事蒙特卡洛渲染工作的人必须参考的标准文本。

电影和动画产业是离线渲染的早期采用者。Pixar 于 1986 年作为 Lucasfilm 计算机部门的分拆公司成立,并开发了 RenderMan——一种后来成为电影行业标准的渲染 API 和着色语言。1995 年完全由计算机图形渲染的 Toy Story 展示了基于物理建模的几何、动画和着色可以为叙事带来什么。此后的每一部 Pixar 电影都推动了技术前沿,而 Pixar 的 SIGGRAPH 论文在 1990 年代和 2000 年代一直是这个领域最有影响力的论文之一。

实时图形学革命来自 GPU。最早的消费级显卡用固定硬件处理变换和光栅化;到 1990 年代末,纹理映射、光照和基础效果已经可以由硬件加速。NVIDIA 的 GeForce 256 于 1999 年在硬件中执行变换和光照计算,取代了此前限制实时图形的 CPU 计算。2001–2002 年,随着 DirectX 8 和后续 shader model 版本引入可编程顶点着色器和像素着色器,GPU 从固定功能流水线变成了面向图形的通用并行计算机。这使实时视觉效果爆发式增长:动态阴影、法线贴图、环境光遮蔽、复杂材质效果——这些都作为在 GPU 上运行的着色器程序实现。

基于物理的渲染(physically-based rendering,PBR)运动把离线渲染的严谨性带入了实时图形学。Matt Pharr、Greg Humphreys 和 Wenzel Jakob 的 Physically Based Rendering: From Theory to Implementation(PBRT)于 2004 年首次出版,它以易懂方式处理蒙特卡洛渲染,使更广泛的共同体能够理解这一领域。2000 年代 GPU 的可编程性,使在实时着色器中实现具有物理基础的材质模型成为可能。Walt Disney Animation Studios 于 2012 年发表了他们的材质模型,确立了 “Disney PBR” 方法——一种在物理上可信、同时为艺术控制而参数化的 BRDF(双向反射分布函数)——它在 2010 年代成为游戏引擎和实时图形学的标准。

NVIDIA 的 RTX 20 系列 GPU 于 2018 年引入用于光线追踪加速的专用硬件(RT cores),第一次使实时光线追踪在实践中可行。Microsoft 的 DirectX Raytracing API 和 Vulkan 光线追踪扩展提供了编程接口。到 2020 年代初,混合渲染——用光栅化处理主要可见性 pass,用光线追踪处理反射、阴影和环境光遮蔽——成为 AAA 游戏的标准方法。Unreal Engine 5 的全局光照系统 Lumen 以及类似系统证明,在消费级硬件上实现生产质量的实时全局光照已经可行。

神经渲染革命始于 Ben Mildenhall 及其同事 2020 年的 Neural Radiance Fields(NeRF)论文。NeRF 把一个场景表示为神经网络:给定一个三维位置和观察方向,网络预测该点的密度和颜色。渲染一个新视角,需要从相机出发沿光线对网络进行积分——计算昂贵,但可以在没有显式几何重建的情况下,从一组输入照片中产生照片级真实结果。这一方法改变了三维重建,并立刻催生了后续工作:更快的变体(Instant NGP,2022)、更高质量,以及对动态场景的扩展。Gaussian Splatting(Kerbl 等,2023)用三维高斯替代神经网络,实现了神经表示的实时渲染速度——这是一个重大进展,使类似 NeRF 的表示在交互式使用中变得实际可行。

生成模型革命——扩散模型根据文本提示生成照片级图像和视频——本身并不主要是图形学发展,但已经成为图形学关切。AI 生成纹理、AI 辅助资产创作、蒙特卡洛渲染中的 AI 降噪(NVIDIA 的 DLSS 和 Intel 的 XeSS 使用神经网络放大低分辨率渲染结果),以及把 AI 生成图像作为神经渲染的训练数据——这些都是生成模型与图形学流水线之间的交叉,正在重塑工作流。

渲染、表示与实时性能

渲染流水线:从场景到像素

渲染是从场景描述中计算图像。两类方法占主导地位:光栅化和光线追踪。它们的差异在于如何确定场景元素与图像像素之间的关系。

光栅化把几何体(三角形)投影到图像平面上,并确定每个三角形覆盖哪些像素。对于每个被覆盖的像素,片元着色器程序会计算颜色,通常使用材质属性、表面法线和光照。光栅化非常高效,因为投影和覆盖计算高度可并行化,并且天然映射到 GPU 架构上。GPU 可以在每一帧中,通过数千个并行着色器单元处理数百万个三角形。限制在于,全局效果——准确反射、折射、全局光照——都需要显式近似,因为光栅化流水线只能看见从相机可见的几何体。

光线追踪从相机向场景中投射光线,并计算它们与几何体的交点。对于每个交点,可以继续投射次级光线(反射光线、阴影光线、折射光线)来计算全局效果。优势是物理准确性:反射、折射、焦散和软阴影都会自然地从光线追踪计算中出现。劣势是成本:向任意场景几何中追踪光线,需要高效空间数据结构(BVH、k-d trees),而每条次级光线还可能产生更多光线,使计算呈树状展开,难以高效并行化。GPU 光线追踪使用专用硬件加速 BVH 遍历和光线—几何相交计算。

生产渲染使用路径追踪:通过随机采样光线路径,追踪光在场景中的传播路径。一条相机光线击中表面;新的光线从材质的 BSDF(双向散射分布函数)中采样出来;这条光线击中另一个表面,如此继续,直到路径逃离场景,或通过 Russian Roulette 终止。正在计算的积分——渲染方程——通过对许多这样的路径样本取平均来估计。路径追踪是无偏的:只要样本足够多,它会收敛到正确答案。方差,也就是图像中的噪声,会随样本数 n 以 1/√n 下降,因此干净图像需要大量样本。

Kajiya 推导出的基于物理的渲染方程,把表面点 x 沿方向 ω_o 发出的辐射亮度 L_o 表达为:

L_o(x, ω_o) = L_e(x, ω_o) + ∫_{Ω} f_r(x, ω_i, ω_o) L_i(x, ω_i) (ω_i · n) dω_i

其中 L_e 是自发辐射亮度,f_r 是 BRDF,L_i 是来自方向 ω_i 的入射辐射亮度。蒙特卡洛路径追踪通过从半球 Ω 中采样入射方向来估计这个积分。BRDF f_r 编码了材质的外观——也就是它如何在不同角度反射光。

几何表示

在场景能够被渲染之前,它必须先被表示出来。不同表示支持不同操作,也有不同效率权衡。

三角网格是实时渲染中的通用格式:由共享顶点和法线的一组三角形组成,并带有额外属性,例如纹理坐标和用于材质的切线。GPU 流水线已经针对三角网格光栅化优化。网格紧凑、易于并行处理,并且被每一种渲染 API 支持。其限制在于,表示平滑表面需要许多三角形,而编辑网格拓扑很不方便。

参数曲面(NURBS、细分曲面)显式表示平滑曲面,被用于建模和电影渲染,其中几何体会在渲染时被细分成三角形。细分曲面是 Pixar 风格角色制作中的标准:一个粗糙控制网格按照规则被细分,生成一个平滑极限曲面,从而在不需要显式指定的情况下提供精确的平滑几何。

有符号距离场(signed distance fields,SDFs)把几何表示为体积中每一点到表面的有符号距离。正值在外部,负值在内部,零值位于表面上。SDF 支持高效 ray marching——沿一条光线前进,每一步长度等于当前位置的 SDF 值,并保证不会错过表面——它被用于程序化几何、字体渲染、碰撞检测和软体仿真。神经有符号距离场是 NeRF 家族的一部分,用神经网络表示隐式曲面。

神经表示(NeRF、Gaussian Splatting)直接把场景表示为一个从位置和观察方向映射到外观的函数。NeRF 使用神经网络;Gaussian Splatting 使用一组具有位置、尺度、旋转、颜色和不透明度的三维高斯。这些表示可以从照片中获取,而不是由人工建模,从而可以对真实场景进行照片级重建;但它们不易编辑,并且依赖新视角合成,而不是显式几何操作。

GPU 架构与实时性能

GPU 在图形学中的优势来自大规模并行性:数千个简单处理器同时在不同数据上执行同一个着色器程序(SIMT,Single Instruction Multiple Threads)。顶点着色器在每个顶点上运行;片元着色器在每个被覆盖的片元上运行;计算着色器则在一组派发的工作项上运行。效率取决于能否让所有这些处理器都忙于一致的工作。

内存层级和计算单元同样重要。GPU 内存带宽通常远高于 CPU 带宽,它决定 GPU 能以多快速度读取纹理和顶点数据。L1 和 L2 缓存可以减少一致访问模式下的带宽压力。纹理采样器,以及带有双线性和三线性插值的硬件纹理缓存,会加速片元着色器中最常见的内存访问模式。

实时渲染中的性能瓶颈,会按照流水线中限制吞吐量的阶段分类。受顶点限制的应用,瓶颈在顶点处理速率;修复方法是减少顶点数量或简化顶点着色器。受片元限制的应用,瓶颈在片元着色器执行;减少 overdraw(像素被多次绘制)或简化片元着色器会有帮助。受带宽限制的应用,瓶颈在纹理或几何内存访问;降低纹理分辨率、压缩纹理,或改善空间局部性会有帮助。识别具体瓶颈需要使用 GPU 专用分析工具,例如 RenderDoc、NSight 和 PIX。

细节层次(level of detail,LOD)系统通过为远离相机的物体使用简化版几何和材质来管理场景复杂度,因为这些距离上的细节不可见。Nanite(Unreal Engine 5 的虚拟化几何系统)把这一点推到极端:任意多边形数量的网格会按需流式传输,并且只按屏幕像素所需分辨率进行光栅化,从而消除了手工制作多个 LOD 层级的需要。

学习这一部分会改变什么

计算机图形学会改变实践者观看图像的方式,也会改变他们对计算能够创造什么的理解。

第一个变化,是关于渲染的视觉素养。学过基于物理的渲染的实践者,可以看着一张渲染图像并分析它:这种光照是否与声称的光源位置一致?这种材质在物理上是否可信?这个噪声模式由什么采样伪影造成?这个阴影在几何上是否正确?这种分析能力——把图像看作物理仿真的输出,而不只是审美选择——不是非实践者自然会发展出来的;它对生产工作和评估 AI 生成图像都有价值。

第二个变化,是 GPU 编程直觉。学习实时图形学之后,实践者会理解 GPU 上哪些操作便宜、哪些操作昂贵,哪些内存访问模式高效、哪些低效,如何阅读 GPU 性能分析并识别瓶颈。这些知识会迁移到图形学之外的任何 GPU 加速计算中——机器学习训练和推理、科学仿真、并行数据处理。

第三个变化,是空间推理的数学词汇。变换、投影、坐标系、齐次坐标、四元数、重心坐标——这些数学工具在计算机科学和工程中都有广泛应用,但它们在图形学语境中最容易被自然习得,因为其视觉意义非常具体。一个亲手实现过投影矩阵的实践者,会以仅仅调用过它的人所没有的方式理解它正在做什么。

第四个变化,是理解视觉外观的物理基础。基于物理的渲染要求理解光如何与材质相互作用——吸收、散射、折射、衍射。这种物理理解不仅对渲染有用,也有助于更广泛地评估视觉主张:一张照片是否符合其声称的光照条件,一个场景中的材质是否可信,一张 AI 生成图像是否在物理上自洽。

资源

书籍与文本

Pharr、Jakob 和 Humphreys 的 Physically Based Rendering: From Theory to Implementation(PBRT,第 4 版,2023,可在 pbr-book.org 免费获取)是离线渲染中最重要的单一文本。它既提供数学理论——渲染方程、蒙特卡洛积分、BSDF 模型——也提供一个完整参考路径追踪器实现,代码精心编写且注释充分。阅读 PBRT 并实现其中部分内容,是理解生产渲染的决定性路径。第四版覆盖了当代发展,包括 GPU 渲染和体积散射。该书可在线免费获取。

Akenine-Möller、Haines、Hoffman、Pesce、Iwanicki 和 Hillaire 的 Real-Time Rendering(第 4 版,2018,realtimerendering.com)是实时图形学的经典参考书。它覆盖 GPU 流水线、着色模型、全局光照近似、抗锯齿、纹理,以及几十种具体技术,并带有丰富文献引用。它过于全面,不适合按顺序通读;但对于理解任何具体技术,以及理解技术在更广泛图景中的位置,非常有价值。

Hughes、van Dam、McGuire、Sklar、Foley、Feiner 和 Akeley 的 Computer Graphics: Principles and Practice(第 3 版,2013)是综合性基础教材,以数学深度覆盖计算机图形学的完整范围。它比 PBRT 或 RTR 更学术,在当代 GPU 中心实践的处理上略显过时,但对于建立数学基础来说仍无可替代。

专门用于数学准备的材料是 Lengyel 的 Mathematics for 3D Game Programming and Computer Graphics(第 3 版,2011),它覆盖线性代数、三角学、微积分,以及图形学中使用的具体数学结构,例如四元数、齐次坐标和相交算法。它没有独立数学教材那样全面,但更直接适用于图形学工作。

Gregory 的 Game Engine Architecture(第 3 版,2018)是游戏引擎系统的经典教材,覆盖渲染流水线、物理、动画、音频、输入系统,以及它们如何整合。对于追求游戏开发的实践者,它弥合了 PBRT 和 RTR 中图形学概念与生产级游戏引擎工程现实之间的距离。

书籍 作用 类型
Pharr, Jakob & Humphreys, Physically Based Rendering(第 4 版,免费) 离线渲染经典文本 + 实现 入门
Akenine-Möller et al., Real-Time Rendering(第 4 版) 实时渲染经典参考 参考
Hughes et al., Computer Graphics: Principles and Practice(第 3 版) 综合基础教材 参考
Lengyel, Mathematics for 3D Game Programming and Computer Graphics 图形学数学 入门
Gregory, Game Engine Architecture(第 3 版) 游戏引擎系统 深入
Botsch et al., Polygon Mesh Processing(免费) 几何处理 深入
Bridson, Fluid Simulation for Computer Graphics(第 2 版) 物理仿真 深入
Munzner, Visualization Analysis and Design 信息可视化 入门
Veach, PhD thesis(免费) 蒙特卡洛渲染基础 深入
The Graphics Codex(免费样章;完整 app/访问可能付费) 图形学数学、渲染和实现细节的紧凑参考 参考

课程与讲座

Stanford CS148(Introduction to Computer Graphics and Imaging)和 CS348b(Image Synthesis)都有免费材料,覆盖渲染、着色和光线追踪的数学与算法基础。

UC Berkeley CS184/284(Computer Graphics)提供强课程和免费材料,覆盖光栅化、光线追踪、建模和动画,并包含实现项目。

Cem Yuksel 的 YouTube 讲座系列(免费)是最全面的免费计算机图形学视频入门,覆盖着色、渲染、光线追踪和 GPU 编程,并提供清楚的数学解释。

The Book of Shaders(Patricio Gonzalez Vivo 和 Jen Lowe,thebookofshaders.com,免费)是交互式 shader 编程入门——以 GLSL 片元着色器为中心,并提供可在浏览器中运行的视觉示例。它是初学者进入 GPU 编程的最佳入口。

LearnOpenGL(learnopengl.com,免费)是 OpenGL 的标准教程网站,覆盖渲染流水线、着色器、纹理、光照和更高级主题。它是实现图形程序的合适实践入口。

课程 平台 类型
Stanford CS148/CS348b(免费) Stanford course site 入门
UC Berkeley CS184/284(免费) Berkeley course site 入门
Cem Yuksel YouTube lectures(免费) YouTube 实践
The Book of Shaders(免费) thebookofshaders.com 实践
LearnOpenGL(免费) learnopengl.com 实践
Scratchapixel(免费) scratchapixel.com 入门
WebGPU Fundamentals(免费) webgpufundamentals.org 实践
Vulkan Tutorial / Khronos Vulkan Learn(免费) vulkan-tutorial.com / vulkan.org 实践

实践、工具与当前资料

PBRT 源代码(可在 github.com/mmp/pbrt-v4 免费获取)是 PBRT 教材配套实现。编译并运行它,然后修改具体 integrator 或 BSDF 实现,是渲染中最集中的学习练习。代码是示范级 C++,展示了生产质量渲染基础设施如何组织。

RenderDoc(renderdoc.org,免费)是图形调试中首选 GPU 调试器——它可以捕获一帧,让你逐步查看 draw calls,并显示每个阶段中每个 render target 和 resource 的状态。对任何图形应用使用它,都能立刻以具体形式揭示渲染流水线。

Shadertoy(shadertoy.com,免费)是一个在浏览器中编写片元着色器的环境。社区已经创建了数千个 shader,展示了从基础 ray marching 到完整路径追踪器的各种图形技术。阅读和修改社区 shader,是学习实践性 shader 编程最高效的方式。

Blender(blender.org,免费)是专业三维创作套件,包含路径追踪渲染器 Cycles、实时 viewport 渲染器 Eevee、建模工具和 Python 脚本。它的源代码开放且组织良好;一边使用它作为渲染引擎,一边阅读 PBRT,可以澄清理论和实现之间的连接。

Mildenhall 等人的 NeRF 论文(2020,可在 arxiv.org 免费获取)和 Kerbl 等人的 3D Gaussian Splatting 论文(2023,可在 arxiv.org 免费获取)是神经渲染的一手来源。配套代码也在 GitHub 上发布。完整阅读 NeRF 代码,是进入神经渲染的入口。

资源 平台 类型
PBRT source code(免费) github.com/mmp/pbrt-v4 实践
RenderDoc(免费) renderdoc.org 实践
Shadertoy(免费) shadertoy.com 实践
Blender(免费) blender.org 实践
Mildenhall et al., NeRF code(免费) GitHub 实践
Kerbl et al., Gaussian Splatting code(免费) GitHub 实践
SIGGRAPH proceedings(延迟后免费) dl.acm.org 参考
Inigo Quilez’s website(免费) iquilezles.org 实践

陷阱

陷阱 为什么会误导 更好的回应
把图形学当成 API 调用 图形 API(OpenGL、Vulkan、DirectX、游戏引擎 API)提供了进入 GPU 流水线的入口,但 API 不是这门学科本身——渲染算法和数学基础才是。只学 API 而不学底层数学的实践者,无法推理 API 正在做什么,无法从源头诊断渲染伪影,也无法把技术适配到新问题上。 先学习数学和算法,再学习 API。先用软件实现一个简单光线追踪器,再用 OpenGL 实现光栅化器。理解 GPU 正在计算什么,会让 API 变得可理解;只理解 API,则会让计算保持不透明。
跳过数学 图形学依赖线性代数(变换、投影)、微积分(渲染中的积分)和概率(蒙特卡洛采样)。逃避数学的学习者会很快遇到天花板——他们可以照着教程实现,但无法扩展。PBRT 从第一性原理推导一切;花时间理解推导,而不是直接跳到代码,是获得持久能力的投资。 在渲染算法之前,先掌握数学基础。§2.3 中的线性代数、§2.4 中的微积分和 §2.5 中的概率都是直接承重的。PBRT 前三章是理解渲染需要哪些数学的合适起点。
把离线渲染和实时渲染当成彼此分离的主题 许多学习者会专注于实时渲染(GPU、游戏引擎、60fps)或离线渲染(路径追踪、电影、每帧数小时),并把它们当成无关领域。事实上,它们共享数学基础(渲染方程、BRDF、光传输),而现代实时渲染也越来越多地使用光线追踪技术。 同时学习两个传统。用 PBRT 学习离线渲染基础,用 RTR 学习实时渲染。只有理解二者,才能看懂当代结合光栅化和光线追踪的混合渲染流水线。
把神经渲染当成古典渲染的替代品 神经渲染方法(NeRF、Gaussian Splatting)在从照片进行新视角合成方面有惊人能力,但它们并不是古典渲染的通用替代品:它们不容易支持动态场景,不容易显式编辑几何,而且首先需要输入照片。放弃古典渲染、转向神经方法的实践者,在试图把它们用于生产时会发现这些限制。 先学习古典渲染,再把神经渲染作为扩展。理解路径追踪正在计算什么,会让 NeRF 的替代方案——把场景表示成一个由路径追踪解释的神经函数——变得可读。没有古典基础,神经渲染就是黑箱。
避免使用 GPU 调试工具 图形 bug 是视觉性的——颜色错误、几何缺失、着色不正确、纹理损坏。诊断这些问题需要看见 GPU 在每个阶段到底计算了什么,这就需要 GPU 调试工具(RenderDoc、NSight)。用 print 语句和猜测调试图形程序的实践者,花费的时间会是那些在 RenderDoc 中捕获一帧、检查每个 render target,并找到错误具体出现阶段的实践者的十倍。 在 GPU 编程一开始就学习 RenderDoc,而不是在没有它的情况下挣扎数月之后再学。捕获任何图形应用的一帧——哪怕只是一个教程——然后逐步查看 draw calls。GPU 流水线会以阅读无法提供的方式变得具体、可读。
把视觉输出当成二元的正确/错误 渲染图像不是二元的——它有质量程度,有具有具体原因的可见伪影,也有不同质量维度之间的权衡。把渲染当成“看起来还行”或“坏了”的实践者,无法系统性提高质量。 通过批判性分析渲染图像来发展视觉素养:噪声模式是什么,由什么造成,换一种采样器会如何改变?有哪些锯齿伪影?环境光遮蔽在物理上是否可信?阅读 RTR 中关于伪影的章节,然后在渲染图像中寻找这些伪影,会培养生产图形学所需的批判眼光。