Two ceilings, one runtime

Any kernel has two independent lower bounds on its execution time. It cannot finish before it has performed its arithmetic at peak rate, and it cannot finish before it has moved its operands at peak bandwidth:

t ≥ max( FLOPs / P_peak , Bytes / BW )

P_peak = peak FLOP/s      BW = peak byte/s

Whichever term wins names the regime: if FLOPs / P_peak dominates the kernel is compute-bound, if Bytes / BW dominates it is bandwidth-bound. Divide the two workload quantities and you get the arithmetic intensity I = FLOPs / Bytes, in FLOP per byte. Divide the two machine quantities and you get the machine balance B_m = P_peak / BW. The test is then trivial: I < B_m means bandwidth-bound. Everything below is arithmetic on those two numbers.

Advertisement

Prefill and decode are different machines

The same weights land on opposite sides of that inequality depending on how many tokens are in flight. During prefill the prompt’s N tokens are processed together, so a projection is a real matrix-matrix product: X: [N, d] times W: [d, d], and each weight is loaded once and reused across all N rows.

During decode there is exactly one new token per sequence, so the same projection becomes x: [1, d] times W: [d, d] — a matrix-vector product, a GEMV. The arithmetic collapsed by a factor of N but the weight traffic did not change: you still stream every parameter of the layer through the memory hierarchy to produce one token. Prefill is compute-bound and scales with FLOPs; decode is bandwidth-bound and scales with bytes.

Advertisement

The arithmetic intensity of a matrix-vector product

Take the decode-time projection y = W x with W: [n, n] in fp16 and x: [n]. Count both sides:

FLOPs = 2 * n^2          (one multiply + one add per weight)
Bytes = 2 * n^2          (n^2 weights × 2 bytes, fp16)
      + 2 * n   (read x)
      + 2 * n   (write y)

I = 2n^2 / (2n^2 + 4n) → 1 FLOP/byte   as n grows

One FLOP per byte — exactly, in the limit, because fp16 spends two bytes to fetch a weight that buys you two flops. The activation traffic is O(n) against O(n^2) of weight traffic and vanishes. Contrast the batched case: with B sequences decoding together, X: [B, n], the FLOPs become 2Bn^2 while the weight bytes stay at 2n^2, so I ≈ B. Intensity in decode is essentially the batch size.

Machine balance and the ridge point

Now the hardware side. An H100 SXM delivers roughly 495 TFLOP/s of dense bf16 tensor-core throughput against 3.35 TB/s of HBM3 bandwidth (the quoted 989 TFLOP/s assumes 2:1 structured sparsity, which inference rarely uses):

B_m = 495e12 / 3.35e12 ≈ 148 FLOP/byte   (H100, dense bf16)
B_m = 312e12 / 2.04e12 ≈ 153 FLOP/byte   (A100 80GB, bf16)

This threshold is the ridge point of the roofline, and it has climbed for a decade because arithmetic throughput grew far faster than DRAM bandwidth. A GEMV at I ≈ 1 is therefore not slightly below the ridge, it is two orders of magnitude below it. Expect roughly 1/148 of peak FLOPs during decode — the machine working perfectly, not a bug.

Bytes per token: the decode budget

Because decode is bandwidth-bound, its speed limit is a division. Every generated token requires reading every weight the forward pass touches:

bytes_per_token ≈ P * b        P = parameters, b = bytes/param
t_token       ≈ bytes_per_token / BW
tok/s         ≈ BW / (P * b)

Worked example. A 7B model in fp16 is 7e9 × 2 = 14 GB. On an H100 at 3.35 TB/s:

t_token ≈ 14e9 / 3.35e12 = 4.2 ms  → ~239 tok/s  (batch 1, ceiling)

compute cross-check:  2P = 14 GFLOP
t_compute ≈ 14e9 / 495e12 = 28 µs   — 150× faster

The arithmetic takes 28 microseconds; fetching the operands takes 4.2 milliseconds. Real systems land at 60–80% of that ceiling, but the ceiling itself follows from two numbers you can look up — which is what makes decode latency so unusually predictable.

The KV cache: bandwidth that grows with context

Weights are not the only bytes. Attention at decode must read the entire key/value cache for every sequence in the batch. Per token per layer:

kv_bytes = 2 * n_kv_heads * d_head * b_kv     (2 = K and V)

Llama-2-7B, MHA:  2 * 32 * 128 * 2 = 16 KB/layer × 32 = 512 KB/token
Llama-3-8B, GQA:  2 *  8 * 128 * 2 =  4 KB/layer × 32 = 128 KB/token

At batch B and context L, attention reads B × L × kv_bytes per step — and unlike the weights, that traffic is not amortized across the batch, because each sequence has its own cache. For the 7B MHA model at B = 32, L = 4096 that is 32 × 4096 × 512 KB ≈ 64 GB per step against 14 GB of weights — the dominant term has flipped from weights to cache. That is why grouped-query attention and KV quantization exist: they are bandwidth optimizations wearing an architecture costume.

The batch-size crossover

Since I ≈ B for the weight matmuls, the naive crossover into compute-bound territory is B ≈ B_m, i.e. batch sizes in the low hundreds on current accelerators. Below it, doubling the batch is nearly free: you pay the same weight traffic and get twice the tokens, so throughput scales almost linearly while per-token latency barely moves. Above it the tensor cores saturate and batching costs latency.

Two caveats keep this from being one clean number. First, KV traffic grows with B × L, so at long context the memory term never stops growing and the crossover drifts upward — no batch size makes a 32k-context workload compute-bound. Second, memory capacity usually binds first: those caches have to fit in HBM. Hence the effort serving stacks spend on continuous batching and paged KV caches — batching is the only lever that moves decode intensity by an order of magnitude.

Elementwise ops are bandwidth-bound by construction

Everything that is not a matmul — LayerNorm and RMSNorm, GELU and SiLU, softmax, residual adds, RoPE — touches each element a constant number of times and performs a handful of flops on it. Their intensity is O(1) FLOP/byte regardless of shape or batch size, which puts them permanently on the memory roof, prefill included.

A separate GELU kernel on an [N, 4d] fp16 tensor reads 8Nd bytes and writes 8Nd bytes for a few flops per element; run norm, activation and residual as three launches and you pay three round trips to HBM for free arithmetic. The fix is fusion — keep the intermediate in registers and emit one kernel. Same principle as FlashAttention, which never materializes the [N, N] attention matrix at all.

Quantization is a bandwidth optimization

Look again at tok/s ≈ BW / (P × b). With BW fixed by the hardware and P by the model, the only remaining lever is b. Halving precision halves the traffic and roughly doubles the decode ceiling:

7B model, H100 @ 3.35 TB/s, batch 1

fp16  b=2    14.0 GB  →  4.2 ms  →  ~239 tok/s
int8  b=1     7.0 GB  →  2.1 ms  →  ~479 tok/s
int4  b=0.5   3.5 GB  →  1.0 ms  →  ~957 tok/s

Note what this explains: weight-only quantization speeds up decode even when the matmul still runs in fp16 after dequantization, because the win was never in the arithmetic. It also bounds the trick — below 4 bits, activation and KV traffic take over — and it does nothing for a compute-bound prefill.