Autoregressive generation splits into two phases with opposite personalities. Prefill chews through the whole prompt in one parallel pass and is compute-bound; decode then emits the answer one token at a time, and it is a different beast entirely. Each decode step reads the entire model from memory — every weight, plus the full KV cache — only to produce a single token. The arithmetic it performs on all those bytes is tiny, so the GPU spends most of its time waiting on memory, not computing. This article stays on decode (its sibling covers prefill math) and works through why it is memory-bandwidth-bound, the per-token latency formula, a worked example, why batching is the one big throughput lever, the MBU metric that scores how well you are doing, and the disaggregation idea that stops prefill and decode from fighting over the same GPU.
Decode is a strictly serial loop
Prefill is embarrassingly parallel: given a prompt of N tokens, the model processes all N positions at once, filling the GPU with a big matrix-multiply and populating the KV cache for every layer. Decode cannot do that. Token t+1 depends on token t, which the model has not produced yet, so generation is an inherently sequential loop: run a full forward pass, sample one token, append it, run another full forward pass, and repeat.
The critical detail is what each of those passes actually computes. Thanks to the KV cache, the model does not re-process the whole context — the keys and values of every previous token are already stored. So a decode step feeds exactly one new token through the network. Every weight matrix is multiplied by a single vector (a [1, d] activation), not by an [N, d] batch. That is a matrix–vector product, and matrix–vector products are the classic memory-bound workload: you touch each weight once and do almost nothing with it before moving on.
Arithmetic intensity: why decode starves the GPU
Arithmetic intensity (AI) is the ratio that decides whether a kernel is compute-bound or memory-bound: FLOPs performed divided by bytes moved from memory. Consider a weight matrix W with P parameters stored in fp16 (2 bytes each). Multiplying it by one activation vector costs one multiply and one add per parameter — about 2P FLOPs — while reading the matrix costs 2P bytes.
AI = FLOPs / bytes
= 2P / (2P) (fp16 weights, batch = 1)
≈ 1 FLOP per byteAn arithmetic intensity of roughly 1 is the signature of decode. Now compare it to the hardware. An NVIDIA A100 delivers about 312 TFLOP/s of fp16 compute against roughly 2 TB/s of memory bandwidth — a ratio near 156 FLOPs per byte. The roofline ‘ridge point’ sits at 156; a workload needs that much arithmetic intensity just to keep the compute units busy. Decode arrives with an intensity of 1. It is two orders of magnitude below the ridge, so the tensor cores sit ~99% idle while the memory bus does all the work. Decode does not have a compute problem; it has a bandwidth problem.
The per-token latency formula
Because decode is memory-bound, its latency is governed almost entirely by how many bytes must cross the memory bus per step, not by how many FLOPs run. That gives a clean, predictive formula for time-per-output-token (TPOT):
t_token ≈ bytes_moved / memory_bandwidth
bytes_moved = (P × bytes_per_param) # read all weights
+ (KV_cache_bytes) # read the whole cache
tokens_per_sec ≈ 1 / t_tokenEvery decode step must stream all the model weights from high-bandwidth memory (HBM) into the on-chip compute units, plus the full KV cache, since attention at the new position reads every stored key and value. The weight term dominates for short-to-moderate contexts; the KV term grows linearly with sequence length and eventually rivals it. Notice what is absent: the FLOP count. You can estimate the speed of a decoder without knowing anything about its compute throughput — only its size in bytes and the memory bandwidth of the chip it runs on. That is what ‘memory-bandwidth-bound’ means in practice.
A worked example: 7B on an A100
Take a 7-billion-parameter model in fp16 and an A100 with ~2 TB/s of bandwidth. First the weights:
weight bytes = 7e9 params × 2 bytes = 14 GB
t_token ≈ 14 GB / 2000 GB/s = 7 ms
throughput ≈ 1 / 7ms ≈ 143 tokens/sec (theoretical ceiling)Seven milliseconds per token, ~143 tokens/second, is the best a single-stream decoder can do on this hardware — set purely by moving 14 GB per step. Real systems reach perhaps 70–85% of that because no kernel hits peak bandwidth, so ~100–120 tokens/second is a realistic single-sequence number. Now add context. With a typical 7B config the KV cache runs on the order of ~0.5 MB per token; at 4K tokens that is ~2 GB, nudging bytes-moved to ~16 GB and TPOT to ~8 ms. At 32K tokens the cache alone approaches the weight footprint and TPOT climbs accordingly — which is exactly why long contexts decode more slowly even though the per-step compute barely changes.
Batching: the one big throughput lever
Here is the insight that makes LLM serving economical. At batch size 1 you read 14 GB to produce a single token — a catastrophic ratio. But if you decode B sequences together, the weight matrices are read once and reused for all B tokens. The expensive 14 GB weight stream is amortized across the whole batch.
bytes per step = weights(14 GB) + B × KV_per_seq
tokens per step = B
weight cost per token = 14 GB / B # falls as B growsThe matrix–vector product becomes a matrix–matrix product of shape [B, d], and arithmetic intensity rises roughly in proportion to B. Throughput scales almost linearly with batch size while it stays memory-bound — doubling the batch nearly doubles tokens/second at little extra latency — until one of two walls appears: the KV caches fill HBM capacity, or intensity crosses the roofline ridge and decode finally becomes compute-bound. Batching is the throughput knob for decode, which is why serving stacks work so hard (continuous batching, paged attention) to keep the batch large and full.
MBU: model bandwidth utilization
Just as MFU (model FLOPs utilization) scores a compute-bound workload against a chip’s peak FLOP/s, MBU — model bandwidth utilization — scores a memory-bound workload against its peak bandwidth. It is the natural yardstick for decode.
MBU = achieved_bandwidth / peak_bandwidth
= (bytes_per_token × tokens_per_sec) / peak_bandwidthSuppose the 7B decoder streams 14 GB/token and hits 120 tokens/sec on the A100: achieved bandwidth is 14 × 120 = 1680 GB/s, so MBU = 1680 / 2000 ≈ 0.84 — 84%, which is excellent. A low MBU (say 30–40%) signals wasted bandwidth: tiny batches, kernel-launch overhead, poor memory access patterns, or quantization that is not actually reducing bytes read. Because decode latency is bytes-over-bandwidth, MBU is the single most honest health metric for a decode serving system — report it alongside tokens/sec, not instead of it.
Why quantization helps decode specifically
The latency formula also explains why weight quantization is such a direct win for decode. If TPOT is weight_bytes / bandwidth and you halve the bytes per weight, you halve the dominant term. Moving from fp16 (2 bytes) to int8 (1 byte) cuts the 14 GB weight stream to ~7 GB, so the ceiling roughly doubles to ~285 tokens/second on the same A100 — not because arithmetic got cheaper, but because fewer bytes cross the bus.
This is the opposite of the prefill story, where quantization’s main payoff is fitting a bigger model or batch rather than raw speed, since prefill is compute-bound. In decode, bytes moved are the latency, so 4-bit weights (~3.5 GB) can quadruple the bandwidth-limited ceiling. The catch is that the KV cache is not shrunk by weight quantization; at long context the cache term dominates, which is why KV-cache quantization becomes a separate, complementary lever once sequences get long.
Prefill and decode disaggregation
Prefill and decode want different things from a GPU. Prefill is a dense, compute-bound burst that saturates the tensor cores; decode is a long, memory-bound trickle that leaves them idle. Run both on the same device and they interfere: a chunky prefill for one request stalls the steady token stream of every request already decoding, spiking their inter-token latency and blowing tail-latency SLOs.
Disaggregation is the architectural response: run prefill on one pool of GPUs and decode on another, connected by a hand-off that transfers the freshly-built KV cache from a prefill worker to a decode worker. Each pool is then tuned for its own bottleneck — prefill nodes chase FLOP utilization (MFU), decode nodes chase bandwidth utilization (MBU) and large batches — and neither phase pauses the other. Systems like DistServe and Splitwise show this can improve goodput under latency constraints, at the cost of moving KV-cache bytes across the interconnect. It is the clean payoff of taking the two-phase distinction seriously: because prefill and decode are bound by different resources, they are best served by different, separately-scaled hardware.
What it means for CPU and small models
The decode picture explains why CPUs and small language models pair so naturally. A CPU has modest FLOP throughput but, more to the point, modest memory bandwidth — tens to low-hundreds of GB/s versus a GPU’s thousands. Since decode is bandwidth-bound, TPOT on a CPU is still just weight_bytes / bandwidth: a 3B model quantized to 4 bits is ~1.5 GB, so on a machine with ~50 GB/s of usable bandwidth you get on the order of 50 / 1.5 ≈ 33 tokens/second — genuinely usable for a single interactive stream.
Two consequences follow. First, shrinking the model (fewer or smaller weights, aggressive quantization) buys near-proportional decode speed on commodity hardware, which is the whole premise of on-device SLMs. Second, batching’s amortization mostly does not apply to a single-user laptop, so per-token latency there is set by model bytes and bandwidth with little room to hide — making the byte count of your weights the number to optimize. Decode math is the reason a well-quantized small model feels responsive on a CPU while a large one crawls.