Bandwidth-Bound Operations

Two ceilings, one runtime

Any kernel has two independent lower bounds on its execution time. It cannot finish before it has performed its arithmetic at peak rate, and it cannot finish before it has moved its operands at peak bandwidth:

t ≥ max( FLOPs / P_peak , Bytes / BW )

P_peak = peak FLOP/s      BW = peak byte/s

Whichever term wins names the regime: if FLOPs / P_peak dominates the kernel is compute-bound, if Bytes / BW dominates it is bandwidth-bound. Divide the two workload quantities and you get the arithmetic intensity I = FLOPs / Bytes, in FLOP per byte. Divide the two machine quantities and you get the machine balance B_m = P_peak / BW. The test is then trivial: I < B_m means bandwidth-bound. Everything below is arithmetic on those two numbers.

Prefill and decode are different machines

The same weights land on opposite sides of that inequality depending on how many tokens are in flight. During prefill the prompt’s N tokens are processed together, so a projection is a real matrix-matrix product: X: [N, d] times W: [d, d], and each weight is loaded once and reused across all N rows.

During decode there is exactly one new token per sequence, so the same projection becomes x: [1, d] times W: [d, d] — a matrix-vector product, a GEMV. The arithmetic collapsed by a factor of N but the weight traffic did not change: you still stream every parameter of the layer through the memory hierarchy to produce one token. Prefill is compute-bound and scales with FLOPs; decode is bandwidth-bound and scales with bytes.

The arithmetic intensity of a matrix-vector product

Take the decode-time projection y = W x with W: [n, n] in fp16 and x: [n]. Count both sides:

FLOPs = 2 * n^2          (one multiply + one add per weight)
Bytes = 2 * n^2          (n^2 weights × 2 bytes, fp16)
      + 2 * n   (read x)
      + 2 * n   (write y)

I = 2n^2 / (2n^2 + 4n) → 1 FLOP/byte   as n grows

One FLOP per byte — exactly, in the limit, because fp16 spends two bytes to fetch a weight that buys you two flops. The activation traffic is O(n) against O(n^2) of weight traffic and vanishes. Contrast the batched case: with B sequences decoding together, X: [B, n], the FLOPs become 2Bn^2 while the weight bytes stay at 2n^2, so I ≈ B. Intensity in decode is essentially the batch size.

Machine balance and the ridge point

Now the hardware side. An H100 SXM delivers roughly 495 TFLOP/s of dense bf16 tensor-core throughput against 3.35 TB/s of HBM3 bandwidth (the quoted 989 TFLOP/s assumes 2:1 structured sparsity, which inference rarely uses):

B_m = 495e12 / 3.35e12 ≈ 148 FLOP/byte   (H100, dense bf16)
B_m = 312e12 / 2.04e12 ≈ 153 FLOP/byte   (A100 80GB, bf16)

This threshold is the ridge point of the roofline, and it has climbed for a decade because arithmetic throughput grew far faster than DRAM bandwidth. A GEMV at I ≈ 1 is therefore not slightly below the ridge, it is two orders of magnitude below it. Expect roughly 1/148 of peak FLOPs during decode — the machine working perfectly, not a bug.

Bytes per token: the decode budget

Because decode is bandwidth-bound, its speed limit is a division. Every generated token requires reading every weight the forward pass touches:

bytes_per_token ≈ P * b        P = parameters, b = bytes/param
t_token       ≈ bytes_per_token / BW
tok/s         ≈ BW / (P * b)

Worked example. A 7B model in fp16 is 7e9 × 2 = 14 GB. On an H100 at 3.35 TB/s:

t_token ≈ 14e9 / 3.35e12 = 4.2 ms  → ~239 tok/s  (batch 1, ceiling)

compute cross-check:  2P = 14 GFLOP
t_compute ≈ 14e9 / 495e12 = 28 µs   — 150× faster

The arithmetic takes 28 microseconds; fetching the operands takes 4.2 milliseconds. Real systems land at 60–80% of that ceiling, but the ceiling itself follows from two numbers you can look up — which is what makes decode latency so unusually predictable.

The KV cache: bandwidth that grows with context

Weights are not the only bytes. Attention at decode must read the entire key/value cache for every sequence in the batch. Per token per layer:

kv_bytes = 2 * n_kv_heads * d_head * b_kv     (2 = K and V)

Llama-2-7B, MHA:  2 * 32 * 128 * 2 = 16 KB/layer × 32 = 512 KB/token
Llama-3-8B, GQA:  2 *  8 * 128 * 2 =  4 KB/layer × 32 = 128 KB/token

At batch B and context L, attention reads B × L × kv_bytes per step — and unlike the weights, that traffic is not amortized across the batch, because each sequence has its own cache. For the 7B MHA model at B = 32, L = 4096 that is 32 × 4096 × 512 KB ≈ 64 GB per step against 14 GB of weights — the dominant term has flipped from weights to cache. That is why grouped-query attention and KV quantization exist: they are bandwidth optimizations wearing an architecture costume.

The batch-size crossover

Since I ≈ B for the weight matmuls, the naive crossover into compute-bound territory is B ≈ B_m, i.e. batch sizes in the low hundreds on current accelerators. Below it, doubling the batch is nearly free: you pay the same weight traffic and get twice the tokens, so throughput scales almost linearly while per-token latency barely moves. Above it the tensor cores saturate and batching costs latency.

Two caveats keep this from being one clean number. First, KV traffic grows with B × L, so at long context the memory term never stops growing and the crossover drifts upward — no batch size makes a 32k-context workload compute-bound. Second, memory capacity usually binds first: those caches have to fit in HBM. Hence the effort serving stacks spend on continuous batching and paged KV caches — batching is the only lever that moves decode intensity by an order of magnitude.

Elementwise ops are bandwidth-bound by construction

Everything that is not a matmul — LayerNorm and RMSNorm, GELU and SiLU, softmax, residual adds, RoPE — touches each element a constant number of times and performs a handful of flops on it. Their intensity is O(1) FLOP/byte regardless of shape or batch size, which puts them permanently on the memory roof, prefill included.

A separate GELU kernel on an [N, 4d] fp16 tensor reads 8Nd bytes and writes 8Nd bytes for a few flops per element; run norm, activation and residual as three launches and you pay three round trips to HBM for free arithmetic. The fix is fusion — keep the intermediate in registers and emit one kernel. Same principle as FlashAttention, which never materializes the [N, N] attention matrix at all.

Quantization is a bandwidth optimization

Look again at tok/s ≈ BW / (P × b). With BW fixed by the hardware and P by the model, the only remaining lever is b. Halving precision halves the traffic and roughly doubles the decode ceiling:

7B model, H100 @ 3.35 TB/s, batch 1

fp16  b=2    14.0 GB  →  4.2 ms  →  ~239 tok/s
int8  b=1     7.0 GB  →  2.1 ms  →  ~479 tok/s
int4  b=0.5   3.5 GB  →  1.0 ms  →  ~957 tok/s

Note what this explains: weight-only quantization speeds up decode even when the matmul still runs in fp16 after dequantization, because the win was never in the arithmetic. It also bounds the trick — below 4 bits, activation and KV traffic take over — and it does nothing for a compute-bound prefill.

Turning bandwidth into compute: speculation and MoE

If intensity is tokens-per-weight-read, then any trick that produces more than one token per pass is a direct intensity multiplier. Speculative decoding uses a cheap draft model to propose k tokens and verifies all of them in a single forward pass of the target model. Weight traffic is paid once, accepted tokens come out k at a time, and intensity rises toward k — you spend idle FLOPs, which were free, to buy back bandwidth, which was not. Medusa-style heads and lookahead decoding exploit the same slack.

Mixture-of-experts pushes on the other term: only routed experts are read, so bytes per token track the active parameters, which is why a sparse 8×7B model decodes at roughly the speed of a dense 13B one — until a large batch routes to every expert anyway.

On a CPU, the ratio is even harsher

CPU inference is where this arithmetic becomes the entire engineering problem. A dual-channel DDR5-5600 desktop delivers about 2 × 5600 MT/s × 8 B ≈ 90 GB/s, roughly 1/37 of an H100, and bytes per token do not care what chip is reading them:

7B fp16   14.0 GB / 90 GB/s = 156 ms  →   ~6 tok/s
7B int4    3.5 GB / 90 GB/s =  39 ms  →  ~26 tok/s
3B int4    1.5 GB / 90 GB/s =  17 ms  →  ~60 tok/s

Apply the usual 60–80% efficiency factor and those are the numbers you observe from a llama.cpp build. This is the real reason 4-bit quantization is table stakes for local inference, why parameter count matters more than any kernel tuning, and why AVX-512 or AMX support moves the needle less than adding a memory channel. It also means threading past the point where memory saturates buys nothing.

Diagnosing it, and the usual traps

The measurement is a sanity check you can do before any profiler: compute bytes_moved / elapsed_time and compare it to the device’s peak. Above roughly 70% of peak bandwidth the kernel is bandwidth-bound, and the only real fixes are moving fewer bytes or moving them once.

Three traps recur. Reporting MFU during decode — a 1–3% model-FLOPs-utilization figure at batch 1 is the expected result, not a finding; report bandwidth utilization instead. Benchmarking prefill and calling it inference — prompt processing is compute-bound and will happily show 40% MFU while generation crawls. Forgetting that bandwidth is shared — the model competes with every other process for the same DRAM channels. Both ceilings are lower in production than on the spec sheet, and the memory ceiling is the one you will hit.