Why it matters
GPUs power modern AI. Every transformer model is trained and served on GPUs. Understanding GPU architecture is understanding the physical substrate of the AI revolution. It also matters for cost: GPU choice can 10x or shrink your training budget.
The architecture
A modern NVIDIA H100 has 132 SMs, each with 128 CUDA cores plus specialized units (tensor cores, RT cores). Total: 16,896 CUDA cores. Compare to a top-end CPU's ~64 cores.
Each SM runs warps (groups of 32 threads) in SIMT (Single Instruction Multiple Thread) fashion. All threads in a warp execute the same instruction simultaneously on different data.
How it works end to end
Memory hierarchy: registers (fastest, per-thread), shared memory (per-SM), L1/L2 cache, HBM (global memory, slowest, largest). Effective GPU code minimizes global memory access.
Execution model: kernels launch millions of threads. GPU scheduler distributes them to SMs. Each SM runs warps to hide memory latency — if one warp stalls on memory, another runs.
Warp divergence hurts performance. If threads in a warp take different code paths (some go left, some go right), execution serializes. Avoid conditional divergence in warp-critical code.