Why architecture matters here
Kernel fusion matters because it eliminates the wasteful HBM memory traffic that dominates memory-bound GPU operations -- fewer kernels each round-tripping data to HBM -- making it a key optimization for the many memory-bound operations in deep learning. Many GPU operations (especially in deep learning -- elementwise ops, activations, normalizations, and the operations around matmuls) are memory-bound (the HBM traffic dominating, not the arithmetic). Running them as separate kernels means each round-trips its data through HBM (read input, write output -- the intermediates going to HBM between operations) -- wasteful HBM traffic (the intermediates unnecessarily round-tripping). Kernel fusion eliminates this (combining the operations into one kernel, keeping intermediates on-chip -- so the data reads from HBM once and writes once) -- avoiding the wasteful traffic (faster for the memory-bound operations). Since many deep-learning operations are memory-bound (and there are many of them), kernel fusion is a key optimization (widely applicable, significant speedups). For optimizing GPU deep-learning workloads (a major concern), kernel fusion is important, and understanding it (why memory-bound operations benefit, how fusion avoids the HBM traffic) is understanding a key GPU optimization.
The keep-intermediates-on-chip insight is the core mechanism, and it's what avoids the HBM traffic. The problem with separate kernels: each kernel reads its input from HBM and writes its output to HBM -- so for a sequence of operations, the intermediate results round-trip through HBM between every operation (operation 1 writes its output to HBM, operation 2 reads it back from HBM -- the intermediate going to HBM and back, unnecessarily). For memory-bound operations (where the HBM traffic dominates), this is wasteful (the intermediates' HBM round trips being a large fraction of the time). Kernel fusion keeps the intermediates on-chip: by computing the sequence of operations in a single kernel, the intermediate results stay in the fast on-chip memory (registers or SRAM -- not written to HBM) -- so operation 1's output is kept on-chip and directly used by operation 2 (no HBM round trip for the intermediate). The kernel reads the input from HBM once (at the start) and writes the final output to HBM once (at the end) -- with all the intermediates staying on-chip (never touching HBM). This eliminates the intermediate HBM traffic (the wasteful round trips) -- so the fused kernel does far less HBM traffic (just the input read and output write -- not the intermediates) -- much faster for the memory-bound operations. This keep-intermediates-on-chip mechanism (fusing operations so the intermediates stay on-chip, avoiding their HBM round trips) is the core of kernel fusion. Understanding the keep-intermediates-on-chip core (fusing to keep intermediates on-chip, avoiding HBM traffic) is understanding how kernel fusion speeds up memory-bound operations.
And the compilers-do-it-automatically reality is what makes fusion practical, since manual fusion is hard. Manually writing fused kernels (a single kernel computing a sequence of operations, managing the on-chip data) is difficult (low-level GPU programming -- managing registers/SRAM, the fused computation) -- impractical to do for every operation sequence. Modern deep-learning compilers do much of the fusion automatically: they analyze the operation graph (the sequence of operations in the model) and fuse what they can (identifying fusable sequences -- e.g., a matmul followed by a bias add and activation -- and generating a fused kernel for them) -- automatically. torch.compile (PyTorch's compiler), Triton (a language/compiler for GPU kernels -- often used for fused kernels), XLA (used by JAX/TensorFlow -- fuses operations), and TensorRT (for inference -- fuses operations) all do automatic fusion. So you often get fusion automatically (by using these compilers -- e.g., torch.compile fusing your model's operations) -- without manually writing fused kernels. This makes fusion practical (the compilers doing the hard work -- analyzing and fusing automatically) -- so leveraging fusion is often about using the compilers (torch.compile, etc.) rather than manual kernel writing. Understanding that compilers do fusion automatically (torch.compile, Triton, XLA, TensorRT -- analyzing and fusing the operation graph) is understanding how fusion is practically achieved.
The architecture: every piece explained
Top row: the problem and mechanism. The problem: each kernel reads its input from HBM and writes its output to HBM -- so a sequence of separate kernels round-trips the intermediates through HBM. Memory-bound ops: operations where the HBM traffic dominates (not the arithmetic -- e.g., elementwise ops, activations, normalizations) -- where the intermediate HBM traffic is wasteful. Fusion: chaining the operations in one kernel (computing the sequence in a single kernel) -- so the intermediates stay on-chip. Keep data on-chip: the intermediates kept in registers or SRAM (fast on-chip memory -- not written to HBM) -- avoiding the intermediate HBM round trips.
Middle row: the fusion types. Elementwise fusion: fusing elementwise operations (bias add, activation, residual add -- all elementwise, easily fused into one kernel) -- a common, simple fusion. Epilogue fusion: fusing the operations after a matmul (the 'epilogue' -- e.g., adding a bias and applying an activation after the matmul) into the matmul kernel -- so the matmul's output isn't written to HBM and re-read for the epilogue (the epilogue done on the matmul's on-chip output). Reduction fusion: fusing operations involving reductions (softmax, layernorm -- which involve reductions across dimensions -- fused to avoid multiple HBM passes) -- more complex but valuable (softmax/layernorm are common and memory-bound). Compilers: the tools doing automatic fusion -- torch.compile (PyTorch), Triton (kernel language/compiler), XLA (JAX/TF), TensorRT (inference) -- analyzing the operation graph and fusing.
Bottom rows: overhead and limits. Launch overhead too: fusion also reduces the kernel launch overhead (fewer kernels -- fewer launches -- less CPU-side launch overhead) -- a secondary benefit (beyond the HBM traffic). Limits: not everything fuses -- some operations can't be combined (e.g., operations with incompatible structures, or where fusion would be too complex), and fusion has limits (register/SRAM capacity -- a fused kernel's on-chip data must fit) -- so fusion is applied where it's possible and beneficial. The ops strip: profiling (profiling the GPU work -- identifying memory-bound operations and HBM traffic -- to find fusion opportunities -- e.g., via Nsight -- since fusion helps memory-bound ops), compiler use (using the compilers -- torch.compile, etc. -- to get automatic fusion -- the practical way to fuse), and validation (validating the fused result -- confirming the fusion produces the correct result -- and the speedup -- since fusion changes the execution).
End-to-end flow
Trace fusing a matmul epilogue. A model has a linear layer: a matrix multiply (matmul), then adding a bias, then applying an activation (e.g., GELU). Without fusion (separate kernels): the matmul kernel computes the matmul and writes its output to HBM; the bias-add kernel reads that from HBM, adds the bias, writes back to HBM; the activation kernel reads that from HBM, applies the activation, writes back to HBM -- so the intermediate (the matmul output, then the bias-added result) round-trips through HBM twice (wasteful -- these are memory-bound elementwise operations). With epilogue fusion: the matmul kernel is fused with the bias add and activation (the epilogue) -- so the matmul computes its result on-chip, and the bias add and activation are applied to that on-chip result (in the same kernel -- the intermediates staying on-chip) -- and only the final result (after the activation) is written to HBM. So the intermediates (the matmul output, the bias-added result) never touch HBM (kept on-chip) -- avoiding the wasteful HBM round trips (just the final output written) -- much faster (the memory-bound bias/activation done on-chip, no HBM traffic for the intermediates). The epilogue fusion eliminated the intermediate HBM traffic.
The reduction and compiler vignettes show more fusion and the practical means. A reduction case: a layernorm (which computes a mean and variance -- reductions -- then normalizes) -- without fusion, this involves multiple passes over the data (each pass reading/writing HBM -- memory-bound). With reduction fusion, the layernorm is fused into a single kernel (computing the reductions and the normalization in one pass, keeping the intermediate data on-chip) -- avoiding the multiple HBM passes -- faster. The reduction fusion sped up the layernorm. A compiler case: rather than manually writing these fused kernels, the team uses torch.compile (PyTorch's compiler) -- which analyzes the model's operation graph and automatically fuses the fusable operations (the elementwise sequences, the epilogues, the layernorms/softmaxes) -- so they get the fusion automatically (torch.compile generating the fused kernels) -- without manual kernel writing. The compiler provided the fusion automatically.
The launch-overhead and limits vignettes complete it. A launch-overhead case: fusion also reduces the number of kernel launches (the fused kernel replacing several separate kernels -- fewer launches) -- so beyond the HBM traffic savings, the launch overhead (the CPU-side cost per launch) is reduced (fewer launches) -- a secondary benefit (especially for many small operations). A limits case: the team finds some operations don't fuse (e.g., an operation whose structure isn't compatible with the adjacent ones, or where the fused kernel's on-chip data wouldn't fit in the registers/SRAM) -- so fusion is applied where it's possible (the compiler fusing what it can, leaving the rest) -- recognizing that not everything fuses (the limits). The consolidated discipline the team documents: use kernel fusion to eliminate wasteful HBM traffic for memory-bound operations (fusing operation sequences into single kernels -- keeping the intermediates on-chip -- avoiding their HBM round trips), understand the fusion types (elementwise -- bias/activation/add; epilogue -- after matmul; reduction -- softmax/layernorm), leverage the compilers (torch.compile, Triton, XLA, TensorRT -- automatic fusion -- the practical way, rather than manual kernel writing), benefit from the reduced launch overhead (fewer kernels), recognize the limits (not everything fuses -- register/SRAM capacity, incompatible operations), profile to find fusion opportunities (memory-bound operations, HBM traffic), and validate the fused results and speedup -- because kernel fusion eliminates the wasteful HBM memory traffic that dominates memory-bound GPU operations (fusing operations to keep intermediates on-chip), a key optimization for the many memory-bound operations in deep learning, practically achieved via the compilers.