Why the KV cache is the thing worth quantizing

During autoregressive decoding, a transformer never recomputes attention over the whole prompt for each new token. Instead it caches the key and value vectors it already computed for every past position, and each new token attends against that stored history. That cache is the KV cache, and it grows linearly with sequence length: one key vector and one value vector, per layer, per attention head (or per KV group), for every token seen so far.

The consequence is that on long contexts the KV cache, not the weights, becomes the memory bottleneck. A modest model’s weights are a fixed cost, but a 128k-token context can demand many gigabytes of cache that scale with the conversation. Quantizing the weights helps load the model; quantizing the cache is what lets you serve long contexts and many concurrent sessions without running out of memory. That is why KV quantization is a first-class lever rather than an afterthought.

Advertisement

The bytes-per-token formula

Size the problem before optimizing it. The memory a KV cache occupies is a clean product of five numbers:

bytes = 2 × L × n_kv × d_head × S × bytes_per_elem

  2            → one K tensor and one V tensor
  L            → number of layers
  n_kv         → KV heads (= n_heads for MHA, fewer for GQA/MQA)
  d_head       → dimension per head
  S            → sequence length (tokens cached)
  bytes_per_elem → 2 for fp16, 1 for int8, 0.5 for int4

Every factor except the last is fixed by the architecture and the workload. The last one is the knob quantization turns. Going from fp16 (2 bytes) to int8 (1 byte) halves the cache; int4 (half a byte) quarters it. Because the term is a simple multiplier, the saving is exact and predictable — there is no hidden overhead beyond the small bookkeeping of scales, which we account for below.

Advertisement

A worked memory example

Take a 32-layer model with 8 KV heads, d_head = 128, serving a 32k-token context. Per token the cache holds 2 × 32 × 8 × 128 = 65,536 elements — in fp16, 131,072 bytes, about 128 KB per token.

Across the full 32k context that is 131,072 × 32,768 ≈ 4.3 GB for a single sequence. Switch the cache to int8 and it drops to roughly 2.1 GB; int4 takes it to about 1.1 GB. Multiply by the number of concurrent users and the difference decides how many sessions fit on one machine. The cache scales with S, so doubling the context doubles the cache, and the bytes-per-element knob is often the only cheap way to claw that back.

Integer quantization in one screen

Quantization maps a range of real numbers onto a small set of integers. The standard affine scheme uses a scale and a zero-point:

q     = round(x / scale) + zero_point      # quantize
x_hat = scale × (q - zero_point)         # dequantize

scale      = (x_max - x_min) / (q_max - q_min)
zero_point = q_min - round(x_min / scale)

For int8 the integer range is [-128, 127]; for int4 it is a cramped [-8, 7] — only sixteen levels. The scale sets the step size between representable values and the zero_point shifts the grid so it lines up with the data. On read, attention dequantizes back to a float close to the original. The error each element carries is at most half a step, scale / 2, so everything hinges on keeping the scale small, which means keeping the range you quantize over as tight as possible. That single goal drives every other choice below.

Symmetric vs asymmetric

A symmetric scheme forces the zero-point to zero and centres the integer range on real zero: x_hat = scale × q. It is cheaper (no offset to store or add) and it dequantizes with one multiply, but it wastes half the codes when the data is lopsided — if values run from -0.1 to +3.0, a symmetric range must stretch to ±3.0 and squander the negative side.

An asymmetric scheme keeps the zero-point and fits the range to the true [x_min, x_max], so it spends all its codes where the data actually lives. That precision costs a stored offset per group and a subtract on the read path. The rule of thumb: symmetric when a distribution is roughly centred on zero, asymmetric when it is skewed — another reason keys and values often want different schemes.

Granularity: per-tensor, per-token, per-channel

The other half of quality is how many things share one scale. Coarser sharing is cheaper to store but forces unlike values under one range, inflating the scale.

Per-tensor uses a single scale for the whole K or V tensor — smallest metadata, worst accuracy, because one outlier anywhere blows up the range for everything. Per-token gives each token’s vector its own scale; since attention consumes vectors token by token this is a natural, cheap granularity and the common default for values. Per-channel (per feature dimension) gives each of the d_head channels its own scale, which is the right axis when the trouble lives in specific channels rather than specific tokens. Finer still is group quantization — a scale per block of, say, 64 or 128 elements — the usual compromise that makes int4 viable. More scales means more metadata, so the honest bit-width sits a little above the nominal 4 or 8.

The outlier problem: keys have outlier channels

Here is the fact that makes KV quantization interesting rather than mechanical: the key tensors contain outlier channels. A small number of feature dimensions carry values many times larger in magnitude than the rest, and they sit in the same channels across most tokens. This is the same massive-activation phenomenon seen in weight and activation quantization, and it is poison for a per-token key scale.

Why? A per-token scale must cover the largest value in that token’s vector. If one channel is 20× larger than the others, the scale stretches to fit it and every ordinary channel is then quantized with a step 20× too coarse — they collapse toward a handful of integer levels and lose almost all their information. The fix follows from the diagnosis: quantize keys per-channel (or in small groups along the channel axis) so an outlier channel gets its own generous scale and the well-behaved channels keep their fine one, each confined to its own lane.

Why keys and values want different treatment

Values do not show the same pathology. Value distributions are comparatively smooth and well-behaved across channels, with no systematic outlier dimensions, so a plain per-token scale quantizes them cleanly. Keys, as we just saw, have outliers pinned to specific channels and want a per-channel or channel-group scale.

This is the asymmetric-sensitivity result at the heart of practical KV quantization: keys are harder to quantize than values, and they are hard along a different axis. Push both to int4 with a single naive per-token scheme and the keys degrade first, corrupting attention scores (which pass through a softmax that amplifies small errors) while the values are still fine. The standard recipe therefore treats them separately: values per-token, keys per-channel, and when budgets are tight it is common to keep keys at a higher bit-width than values — int8 keys with int4 values, say — precisely because a bit spent on keys buys more quality than a bit spent on values.