The KV cache is usually introduced as a size — a formula you plug numbers into to see how much memory a sequence eats. That is the right starting point, and the companion article ‘KV cache math — sizing formula, GQA, dtype, paging’ covers it. But sizing is only half the story. To understand why decode is slow, why batch size is capped, and why long context throttles throughput, you have to look one level down: at the actual tensors the cache is made of, how they are laid out in memory, how a token is appended each step, and above all how much data must be moved to generate a single token. We will follow one decode step through the indexing, count the bytes that cross the memory bus, and land on the reason the cache — not compute — decides how many users you can serve at once.
The tensors, not the formula: shapes per layer
Strip away the abstraction and the KV cache is just a pile of ordinary tensors. For every transformer layer, the model keeps two of them: one for keys, one for values. During decode each has the logical shape [batch, heads, seq, head_dim] — often written [B, H, S, D]. B is the number of sequences running concurrently, H the number of key/value heads, S the tokens seen so far, and D the per-head dimension.
So a model with L layers holds 2L such tensors alive at once. The sizing formula everyone quotes — bytes = 2 * L * H * D * S * B * dtype — is simply their total element count times the bytes per element. What it hides is that these are live, addressable buffers read and written on a strict schedule. The one dimension that grows is S: every token you generate extends the sequence axis of all 2L tensors by one. Everything interesting about layout, indexing, and bandwidth follows from that one growing axis.
Layout is a choice: strides and the growing axis
A shape like [B, H, S, D] does not tell you how the numbers sit in memory — that is the layout, and it is a real engineering decision. Memory is one-dimensional; a 4-D tensor is flattened by a set of strides that say how many elements to skip to advance each axis. In the natural row-major [B, H, S, D] layout, D is contiguous (one head’s vector is a solid run of bytes), then you stride by D to move along S, by S·D to change head, and by H·S·D to change sequence.
That choice has consequences. Because S is the axis that grows, a layout that keeps S in the middle means appending a token writes to scattered offsets — one small write per head, strided by D. Some frameworks prefer [B, S, H, D] so that all of a token’s heads land in one contiguous block, making the append a single clean write that plays nicely with block-based allocators. Neither is ‘correct’; kernels are written to match whichever layout the cache uses, chosen to suit how the cache grows and how attention reads it back.
One decode step: indexing and appending
Follow a single generated token. The model has just produced token at position t. In this layer it computes that token’s key vector k_t and value vector v_t, each of shape [B, H, D] — one D-vector per head per sequence. These are appended to the cache at sequence index t: conceptually K[:, :, t, :] = k_t and V[:, :, t, :] = v_t. That write is tiny — it touches only the new slice, 2 · B · H · D elements.
In practice the cache is pre-allocated to the maximum sequence length up front, so no reallocation happens mid-generation; the append writes into an already-reserved slot and a length counter advances. Then attention runs: the fresh query q_t is dotted against all keys K[:, :, :t+1, :], softmaxed, and used to weight all values V[:, :, :t+1, :]. The write is O(1) in sequence length; the read is O(t). That asymmetry is the whole game.
The read dominates: every token touches the whole cache
Here is the fact that makes decode expensive. To generate one token, attention in each layer must read every key and value for every previous token. There is no shortcut: the new query attends to the entire history, so the entire history must be streamed out of memory. The bytes moved per decode step, just for the cache, equal the full current cache size — the whole sizing formula, re-read from scratch every single token.
Compare that to the arithmetic. For each cache element read, attention does only a couple of floating-point operations (a multiply-add for the score, another for the weighted value) — an arithmetic intensity of roughly one op per byte, catastrophically low for an accelerator that can do hundreds of ops per byte of bandwidth. So decode attention is not compute-bound; it is memory-bandwidth-bound. The GPU waits for the cache to arrive over the bus rather than doing math. This is the deep reason the prefill/decode split exists, and why the cache’s size translates so directly into latency.
Worked example: bytes per token and the bandwidth wall
Make it concrete with an 8B-class model using grouped-query attention: L = 32 layers, H = 8 KV heads, D = 128, bf16 (2 bytes). Summing a key and value across all heads and layers gives 128 KiB per token (see below); at an 8K context that is about 1 GiB of cache for one sequence.
per-token KV = 2 * L * H * D * dtype
= 2 * 32 * 8 * 128 * 2 = 128 KiB
cache @ 8K = 8192 * 128 KiB ~ 1 GiB / sequence
read/step = B * 1 GiB (cache) + 16 GB (weights)
step time = bytes_moved / HBM_bandwidth (HBM ~ 3.35 TB/s)On an H100 (≈3.35 TB/s), reading one sequence’s 1 GiB cache costs ≈0.3 ms; the 16 GB of weights cost ≈4.8 ms. At batch 1 the weights dominate. But the weights are read once and shared; the cache is read per sequence. At batch 32, cache traffic is 32 × 1 GiB = 32 GiB — now it dwarfs the weights and sets the pace.
dtype: fewer bytes is less to move, not just less to store
The dtype factor does double duty. Obviously it scales the storage — bf16 is 2 bytes, int8 1, int4 half a byte. Less obviously, and more importantly for speed, it cuts the data movement by the same factor. Since decode is bandwidth-bound, halving the cache dtype roughly halves the per-step cache-read time, so quantizing the KV cache is both a capacity win and a latency win.
The catch is numerical. Keys and values are activations, not weights, and they carry outliers; naive 8-bit or 4-bit quantization can degrade attention scores. Practical schemes quantize per-channel or per-token with scales and dequantize inside the kernel. The trade is favorable: int8 KV typically doubles both the sessions you can hold and the cache-limited throughput for a small, usually acceptable quality cost — which is why KV quantization is one of the highest-leverage knobs in a serving stack.
Contiguous vs paged: fighting fragmentation
Pre-allocating [B, H, S_max, D] per sequence is simple but wasteful: a request that only uses 500 of a reserved 8192 tokens still holds the whole slab, and you cannot pack the leftovers. Because every sequence reserves its maximum, the GPU fills with internal fragmentation and you fit far fewer sessions than the raw memory should allow.
Paged attention (popularized by vLLM) borrows the operating system’s virtual-memory trick. The cache is cut into fixed-size blocks — say 16 tokens each — and a sequence is a list of block pointers rather than one contiguous run. Physically the layout becomes something like [num_blocks, block_size, H, D], and a per-sequence block table maps logical positions to physical blocks. A sequence grabs a free block only when it crosses a boundary, so waste is capped at one partial block instead of a whole reservation. The kernel gains a level of indirection but nearly eliminates fragmentation — often doubling achievable batch size at the same memory, and letting requests share physical blocks for a common prompt prefix.
GQA and MQA: shrinking the head axis
The single most effective way to shrink the cache is to shrink the H in the layout. Multi-head attention gives every query head its own key/value head. Grouped-query attention (GQA) lets several query heads share one key/value head; multi-query attention (MQA) is the limit where all query heads share one KV head. The query tensor keeps its full head count, but the cached tensors only carry H_kv heads.
That is a direct multiplier on both storage and bandwidth. A model with 32 query heads but 8 KV heads — a common 4:1 GQA ratio — stores and reads one-quarter the cache of full multi-head attention, with little quality loss. In our worked example the 8 KV heads (not 32) are exactly why the per-token cost is 128 KiB, not 512 KiB. Because decode is bandwidth-bound, that 4× reduction shows up almost one-for-one as faster decode and 4× more concurrent sessions — which is why GQA is now standard.
The binding constraint: batch size vs context, and the three taxes
Now the payoff. Total accelerator memory is split between weights (fixed) and KV cache (grows with B × S). Weights are paid once and amortized across the whole batch; the cache is a per-sequence, per-token cost that cannot be shared. On an 80 GB card holding 16 GB of weights, roughly 64 GB is left for cache — at 1 GiB per 8K sequence, about 64 concurrent sequences, hard stop.
This creates a direct trade between the two things you might want. You can serve many short conversations or a few long ones, but the product B × S is capped by memory. And throughput has a matching ceiling: batching amortizes the weight read, so tokens-per-second climbs as you add sequences — until cache traffic overtakes weight traffic, after which each new sequence adds its full cache read and the per-token time rises linearly. Beyond that knee, a bigger batch buys little. The KV cache, through both capacity and bandwidth, is the quantity that decides how many users fit and how fast they are served.
It helps to name the three taxes the cache exacts. Capacity: HBM used in proportion to B x S. Bandwidth: read in full every step, making decode memory-bound. Allocation: naive contiguous reservation fragments memory. Every real optimization attacks one of these — GQA/MQA and quantization shrink the bytes, paging kills fragmentation, prefix sharing avoids re-storing common context. They are all just ways of moving fewer bytes, or wasting fewer, along the one axis that grows.