KV cache quantization is the difference between a long-context model that fits in memory and one that does not — but the interesting part is how far you can push it. Naive 8-bit is easy and nearly free; the frontier is 2-bit, where a careless scheme collapses accuracy and a careful one stays almost lossless. The gap between those two outcomes is entirely about structure: keys and values have different statistical shapes, keys hide persistent outlier channels, and the newest tokens matter more than the rest. This piece works through the asymmetric per-channel-key / per-token-value scheme that methods like KIVI use, the outlier analysis that motivates it, group-wise scales, the full-precision residual window that makes per-channel quantization possible during decode, and why the best 2-bit methods need no calibration data at all.
Where the aggression has to come from
The KV cache grows linearly with sequence length and batch size, and at long context it dwarfs the weights. Storing it in fp16 costs 2 × n_layers × 2 × n_kv_heads × d_head bytes per token; int8 halves that, and int8 KV is by now the boring, safe default — quality loss is usually under a point and every serving stack supports it.
The reason people push past int8 is that halving is not enough. A 70B-class model at 32K context spends more on cache than on parameters, and every bit shaved is a proportional increase in the batch size — and throughput — you can fit. 2-bit gives roughly an 8× reduction versus fp16, but a 2-bit integer has only four levels. At that resolution, where you place those four levels, over which slice of numbers, becomes the whole game.
Keys and values are not the same distribution
Affine quantization maps a value onto a b-bit grid with step s = (max - min) / (2^b - 1) and error at most s/2, so a single large value in a group inflates s for everything it shares a scale with. At b = 2, with only 4 levels, one outlier wastes three of them — which is why the founding question is how to group numbers. And the key and value caches answer it differently, because they have different statistics. Look at a layer’s cached tensors, each shaped [n_tokens, d_head], and the asymmetry is stark.
The key cache has a few channels (dimensions of d_head) whose magnitudes are consistently large across every token — fixed columns that light up regardless of input. The value cache shows no such column structure; its large values are scattered. This one difference dictates everything: keys want a scale that isolates the misbehaving columns, values want a scale that follows the tokens. Treating them symmetrically is exactly what falls apart at 2-bit.
The outlier channels: the real enemy at low bits
Those persistent key channels are the crux. Empirically a handful of the d_head dimensions carry values one or two orders of magnitude larger than the rest, token after token. Quantize a key vector per token — one scale across all d_head dimensions — and the outlier channel sets max, blows up s, and every ordinary channel collapses toward zero. At 2-bit the ordinary channels effectively vanish.
The fix is to quantize keys per channel: give each of the d_head columns its own scale, computed over the token dimension. Now an outlier channel gets a large scale that only it pays for, while well-behaved channels keep tight scales and full resolution. Because the outliers are persistent along the same columns, per-channel scales are stable — the outlier structure is a property of the model, not of any one prompt, which is why this needs no per-input tuning.
Per-token values: follow the axis you reduce over
Values get the opposite treatment: quantize per token, one scale across the d_head dimensions of each value vector. The reason is not just that values lack channel outliers — it is how they are consumed. The attention output is a weighted sum over the token axis, out = Σ_t a_t · v_t. Error in one token’s value enters scaled by its attention weight a_t, so keeping each token’s error self-contained keeps the mistakes local and uncorrelated across the sum.
Per-token quantization is also operationally cheap: a value vector is known the moment it is computed, so you quantize it immediately and append it, never needing statistics from other tokens. Values quantize online, one token at a time — a convenience keys do not enjoy, as the residual window will show.
Group-wise quantization: scales at a finer grain
Even per-channel or per-token, one scale over a long axis can be too coarse. Group-wise quantization splits an axis into contiguous groups of size G (commonly 32 or 64), each with its own scale and zero-point: keys chunk each channel into blocks of G tokens; values chunk each token’s d_head into groups of G dimensions.
Group size is the direct memory-versus-accuracy dial. Each group stores side metadata — a scale and a zero-point, typically fp16 — so the true cost is b + (overhead_bits / G) per value. With G = 32 and 16-bit scale plus 16-bit zero-point, ‘2-bit’ is really 2 + 32/32 = 3 effective bits — a real tax you must count. Smaller groups track local ranges better but inflate metadata; larger groups are leaner but coarser. Reporting a bit-width without its group size is meaningless.
The full-precision residual window
Per-channel key quantization has a catch: it needs a scale per column computed over the token axis, yet decoding produces one token at a time, giving one sample per channel — far too few to set a stable scale, and you cannot re-quantize the whole cache each step. KIVI-style schemes resolve this by keeping the most recent R tokens in full precision in a small residual buffer (R a multiple of the group size G, say 128). Only when it fills does a block of G tokens get quantized and flushed into the compressed cache.
This one mechanism does two jobs. It makes per-channel keys tractable — a flushed block carries G samples per channel, enough for a stable scale, and old history is never rescaled. And it protects accuracy where it matters most: the newest tokens, which attention concentrates on, are always read exactly. The residual is a tiny slice of a long context (128 of 32,000 tokens is negligible) yet carries an outsized share of the attention mass, so keeping it exact buys most of the accuracy back for almost no memory.
Putting it together: the 2-bit recipe
Stack the pieces and you get the asymmetric scheme that survives 2-bit. Keys: per channel, group-wise along the token axis, to confine outlier columns. Values: per token, group-wise along d_head, to keep error local to the attention sum. Both: a full-precision residual window of the most recent tokens so decoding stays online and the highest-attention tokens stay exact.
At attention time the compressed blocks are dequantized on the fly inside the kernel and combined with the residual. The headline result: with this exact split, 2-bit KV holds accuracy within a fraction of a point of the fp16 baseline on long-context tasks, while a symmetric 2-bit scheme — same bits, wrong grouping — falls off a cliff. The bits did not change; the structure-awareness did. That is the entire thesis in one comparison.
Calibration-free is a feature, not a shortcut
A striking property of the best low-bit KV methods is that they need no calibration dataset. Weight quantizers like GPTQ or AWQ run a corpus through the model to learn scales; KV quantization computes scales directly from the cached tensors, on the fly, with plain min/max per group — no learned parameter, no offline pass.
This is possible because the scheme respects the structure. Key-channel outliers are an intrinsic, input-independent property of the trained model, so per-channel scales computed from the data at hand are already the right ones — nothing a calibration set would teach that the tensor is not already telling you. It also means the method drops into a serving stack with zero preparation and adapts to any prompt distribution automatically, an edge over schemes that must be re-calibrated when the workload shifts.
A worked memory example
Take a model with n_layers = 32, n_kv_heads = 8, d_head = 128, at 16K context. Per token the fp16 KV cache is 2 (K,V) × 32 × 8 × 128 × 2 bytes = 128 KB; over 16,384 tokens that is 2 GB for one sequence.
Now quantize to 2-bit with G = 32 group-wise scales (fp16 scale + zero-point per group), giving 2 + 32/32 = 3 effective bits, plus a 128-token fp16 residual. The quantized bulk shrinks by 16/3 ≈ 5.3×; the residual adds ~16 MB, negligible against the whole. Net cache drops from 2 GB to roughly 0.38 GB — about a 5× real reduction once metadata is honestly counted. That freed memory converts almost directly into a larger batch, and since decode is bandwidth-bound, into higher throughput.
Why this lands hard on CPU and small models
On a CPU-hosted small language model the calculus is even sharper than on a datacenter GPU. There is no spare VRAM to hide behind — the cache competes for a few gigabytes of system RAM — and decode is bandwidth-bound in the extreme, since the CPU reads the entire cache for every generated token. Shrinking the bytes read per token by 5× is a near-linear speedup on the memory-bound path, not just a capacity win.
The caveat is the dequantization kernel. Every 2-bit value must be unpacked to a float before the dot product, and on a CPU without efficient low-bit unpacking that arithmetic can eat the bandwidth savings. The methods that win fuse dequant into the attention kernel and use SIMD-friendly bit layouts. On CPU, a quantization scheme is only as good as its kernel.
Pitfalls that quietly cost accuracy
The failure modes are specific. Quoting a bit-width without the group size hides the true cost — ‘2-bit’ at G = 32 with fp16 metadata is really 3 effective bits. Symmetric grouping — using per-token scales for keys, or ignoring the channel-vs-token split — is the classic way to make 2-bit collapse; the bits are fine, the axis is wrong. Skipping the residual window quantizes the newest, highest-attention tokens and surrenders the cheapest accuracy you could have kept.
Two more: quantizing the attention sinks — the first few tokens that soak up disproportionate attention — hurts out of proportion to their count, so many schemes keep them exact alongside the residual. And trusting one benchmark: perplexity can look fine while long-range retrieval or exact-copy tasks degrade, since those hinge on recalling a specific distant token that 2-bit may have blurred.