Why it matters
Memory bandwidth is what bounds most GPU workloads. LLM inference is often memory-bandwidth-bound, not compute-bound. Understanding the memory hierarchy determines which optimizations actually help.
The architecture
Registers: fastest, per-thread private, no explicit management. Compiler assigns. Excess register usage reduces active threads per SM (lower occupancy).
Shared memory: fast on-SM SRAM, programmer-managed, 100 KB per SM on modern GPUs. Threads in a block share it. Great for staging data reused across threads.
How it works end to end
L2 cache: shared across the whole GPU, transparent to programmer. Recent NVIDIA GPUs have 50-80 MB L2. Increases effective HBM bandwidth by keeping hot data on-chip.
HBM (High Bandwidth Memory): 40-80 GB on modern GPUs, 2-3 TB/s bandwidth. Physical stacked memory attached to the GPU die.
Coalesced access: when consecutive threads access consecutive addresses, the hardware combines them into one wide load. Uncoalesced patterns can be 10x slower.