Decode is a strictly serial loop

Prefill is embarrassingly parallel: given a prompt of N tokens, the model processes all N positions at once, filling the GPU with a big matrix-multiply and populating the KV cache for every layer. Decode cannot do that. Token t+1 depends on token t, which the model has not produced yet, so generation is an inherently sequential loop: run a full forward pass, sample one token, append it, run another full forward pass, and repeat.

The critical detail is what each of those passes actually computes. Thanks to the KV cache, the model does not re-process the whole context — the keys and values of every previous token are already stored. So a decode step feeds exactly one new token through the network. Every weight matrix is multiplied by a single vector (a [1, d] activation), not by an [N, d] batch. That is a matrix–vector product, and matrix–vector products are the classic memory-bound workload: you touch each weight once and do almost nothing with it before moving on.

Advertisement

Arithmetic intensity: why decode starves the GPU

Arithmetic intensity (AI) is the ratio that decides whether a kernel is compute-bound or memory-bound: FLOPs performed divided by bytes moved from memory. Consider a weight matrix W with P parameters stored in fp16 (2 bytes each). Multiplying it by one activation vector costs one multiply and one add per parameter — about 2P FLOPs — while reading the matrix costs 2P bytes.

AI = FLOPs / bytes
   = 2P / (2P)      (fp16 weights, batch = 1)
   ≈ 1 FLOP per byte

An arithmetic intensity of roughly 1 is the signature of decode. Now compare it to the hardware. An NVIDIA A100 delivers about 312 TFLOP/s of fp16 compute against roughly 2 TB/s of memory bandwidth — a ratio near 156 FLOPs per byte. The roofline ‘ridge point’ sits at 156; a workload needs that much arithmetic intensity just to keep the compute units busy. Decode arrives with an intensity of 1. It is two orders of magnitude below the ridge, so the tensor cores sit ~99% idle while the memory bus does all the work. Decode does not have a compute problem; it has a bandwidth problem.

Advertisement

The per-token latency formula

Because decode is memory-bound, its latency is governed almost entirely by how many bytes must cross the memory bus per step, not by how many FLOPs run. That gives a clean, predictive formula for time-per-output-token (TPOT):

t_token  ≈  bytes_moved / memory_bandwidth

bytes_moved  =  (P × bytes_per_param)      # read all weights
             +  (KV_cache_bytes)             # read the whole cache

tokens_per_sec  ≈  1 / t_token

Every decode step must stream all the model weights from high-bandwidth memory (HBM) into the on-chip compute units, plus the full KV cache, since attention at the new position reads every stored key and value. The weight term dominates for short-to-moderate contexts; the KV term grows linearly with sequence length and eventually rivals it. Notice what is absent: the FLOP count. You can estimate the speed of a decoder without knowing anything about its compute throughput — only its size in bytes and the memory bandwidth of the chip it runs on. That is what ‘memory-bandwidth-bound’ means in practice.

A worked example: 7B on an A100

Take a 7-billion-parameter model in fp16 and an A100 with ~2 TB/s of bandwidth. First the weights:

weight bytes = 7e9 params × 2 bytes = 14 GB

t_token ≈ 14 GB / 2000 GB/s = 7 ms

throughput ≈ 1 / 7ms ≈ 143 tokens/sec   (theoretical ceiling)

Seven milliseconds per token, ~143 tokens/second, is the best a single-stream decoder can do on this hardware — set purely by moving 14 GB per step. Real systems reach perhaps 70–85% of that because no kernel hits peak bandwidth, so ~100–120 tokens/second is a realistic single-sequence number. Now add context. With a typical 7B config the KV cache runs on the order of ~0.5 MB per token; at 4K tokens that is ~2 GB, nudging bytes-moved to ~16 GB and TPOT to ~8 ms. At 32K tokens the cache alone approaches the weight footprint and TPOT climbs accordingly — which is exactly why long contexts decode more slowly even though the per-step compute barely changes.

Batching: the one big throughput lever

Here is the insight that makes LLM serving economical. At batch size 1 you read 14 GB to produce a single token — a catastrophic ratio. But if you decode B sequences together, the weight matrices are read once and reused for all B tokens. The expensive 14 GB weight stream is amortized across the whole batch.

bytes per step   = weights(14 GB) + B × KV_per_seq
tokens per step  = B

weight cost per token = 14 GB / B     # falls as B grows

The matrix–vector product becomes a matrix–matrix product of shape [B, d], and arithmetic intensity rises roughly in proportion to B. Throughput scales almost linearly with batch size while it stays memory-bound — doubling the batch nearly doubles tokens/second at little extra latency — until one of two walls appears: the KV caches fill HBM capacity, or intensity crosses the roofline ridge and decode finally becomes compute-bound. Batching is the throughput knob for decode, which is why serving stacks work so hard (continuous batching, paged attention) to keep the batch large and full.

MBU: model bandwidth utilization

Just as MFU (model FLOPs utilization) scores a compute-bound workload against a chip’s peak FLOP/s, MBU — model bandwidth utilization — scores a memory-bound workload against its peak bandwidth. It is the natural yardstick for decode.

MBU = achieved_bandwidth / peak_bandwidth

    = (bytes_per_token × tokens_per_sec) / peak_bandwidth

Suppose the 7B decoder streams 14 GB/token and hits 120 tokens/sec on the A100: achieved bandwidth is 14 × 120 = 1680 GB/s, so MBU = 1680 / 2000 ≈ 0.84 — 84%, which is excellent. A low MBU (say 30–40%) signals wasted bandwidth: tiny batches, kernel-launch overhead, poor memory access patterns, or quantization that is not actually reducing bytes read. Because decode latency is bytes-over-bandwidth, MBU is the single most honest health metric for a decode serving system — report it alongside tokens/sec, not instead of it.