KV Cache Quantization — 8-bit and 4-bit

Why the KV cache is the thing worth quantizing

During autoregressive decoding, a transformer never recomputes attention over the whole prompt for each new token. Instead it caches the key and value vectors it already computed for every past position, and each new token attends against that stored history. That cache is the KV cache, and it grows linearly with sequence length: one key vector and one value vector, per layer, per attention head (or per KV group), for every token seen so far.

The consequence is that on long contexts the KV cache, not the weights, becomes the memory bottleneck. A modest model’s weights are a fixed cost, but a 128k-token context can demand many gigabytes of cache that scale with the conversation. Quantizing the weights helps load the model; quantizing the cache is what lets you serve long contexts and many concurrent sessions without running out of memory. That is why KV quantization is a first-class lever rather than an afterthought.

The bytes-per-token formula

Size the problem before optimizing it. The memory a KV cache occupies is a clean product of five numbers:

bytes = 2 × L × n_kv × d_head × S × bytes_per_elem

  2            → one K tensor and one V tensor
  L            → number of layers
  n_kv         → KV heads (= n_heads for MHA, fewer for GQA/MQA)
  d_head       → dimension per head
  S            → sequence length (tokens cached)
  bytes_per_elem → 2 for fp16, 1 for int8, 0.5 for int4

Every factor except the last is fixed by the architecture and the workload. The last one is the knob quantization turns. Going from fp16 (2 bytes) to int8 (1 byte) halves the cache; int4 (half a byte) quarters it. Because the term is a simple multiplier, the saving is exact and predictable — there is no hidden overhead beyond the small bookkeeping of scales, which we account for below.

A worked memory example

Take a 32-layer model with 8 KV heads, d_head = 128, serving a 32k-token context. Per token the cache holds 2 × 32 × 8 × 128 = 65,536 elements — in fp16, 131,072 bytes, about 128 KB per token.

Across the full 32k context that is 131,072 × 32,768 ≈ 4.3 GB for a single sequence. Switch the cache to int8 and it drops to roughly 2.1 GB; int4 takes it to about 1.1 GB. Multiply by the number of concurrent users and the difference decides how many sessions fit on one machine. The cache scales with S, so doubling the context doubles the cache, and the bytes-per-element knob is often the only cheap way to claw that back.

Integer quantization in one screen

Quantization maps a range of real numbers onto a small set of integers. The standard affine scheme uses a scale and a zero-point:

q     = round(x / scale) + zero_point      # quantize
x_hat = scale × (q - zero_point)         # dequantize

scale      = (x_max - x_min) / (q_max - q_min)
zero_point = q_min - round(x_min / scale)

For int8 the integer range is [-128, 127]; for int4 it is a cramped [-8, 7] — only sixteen levels. The scale sets the step size between representable values and the zero_point shifts the grid so it lines up with the data. On read, attention dequantizes back to a float close to the original. The error each element carries is at most half a step, scale / 2, so everything hinges on keeping the scale small, which means keeping the range you quantize over as tight as possible. That single goal drives every other choice below.

Symmetric vs asymmetric

A symmetric scheme forces the zero-point to zero and centres the integer range on real zero: x_hat = scale × q. It is cheaper (no offset to store or add) and it dequantizes with one multiply, but it wastes half the codes when the data is lopsided — if values run from -0.1 to +3.0, a symmetric range must stretch to ±3.0 and squander the negative side.

An asymmetric scheme keeps the zero-point and fits the range to the true [x_min, x_max], so it spends all its codes where the data actually lives. That precision costs a stored offset per group and a subtract on the read path. The rule of thumb: symmetric when a distribution is roughly centred on zero, asymmetric when it is skewed — another reason keys and values often want different schemes.

Granularity: per-tensor, per-token, per-channel

The other half of quality is how many things share one scale. Coarser sharing is cheaper to store but forces unlike values under one range, inflating the scale.

Per-tensor uses a single scale for the whole K or V tensor — smallest metadata, worst accuracy, because one outlier anywhere blows up the range for everything. Per-token gives each token’s vector its own scale; since attention consumes vectors token by token this is a natural, cheap granularity and the common default for values. Per-channel (per feature dimension) gives each of the d_head channels its own scale, which is the right axis when the trouble lives in specific channels rather than specific tokens. Finer still is group quantization — a scale per block of, say, 64 or 128 elements — the usual compromise that makes int4 viable. More scales means more metadata, so the honest bit-width sits a little above the nominal 4 or 8.

The outlier problem: keys have outlier channels

Here is the fact that makes KV quantization interesting rather than mechanical: the key tensors contain outlier channels. A small number of feature dimensions carry values many times larger in magnitude than the rest, and they sit in the same channels across most tokens. This is the same massive-activation phenomenon seen in weight and activation quantization, and it is poison for a per-token key scale.

Why? A per-token scale must cover the largest value in that token’s vector. If one channel is 20× larger than the others, the scale stretches to fit it and every ordinary channel is then quantized with a step 20× too coarse — they collapse toward a handful of integer levels and lose almost all their information. The fix follows from the diagnosis: quantize keys per-channel (or in small groups along the channel axis) so an outlier channel gets its own generous scale and the well-behaved channels keep their fine one, each confined to its own lane.

Why keys and values want different treatment

Values do not show the same pathology. Value distributions are comparatively smooth and well-behaved across channels, with no systematic outlier dimensions, so a plain per-token scale quantizes them cleanly. Keys, as we just saw, have outliers pinned to specific channels and want a per-channel or channel-group scale.

This is the asymmetric-sensitivity result at the heart of practical KV quantization: keys are harder to quantize than values, and they are hard along a different axis. Push both to int4 with a single naive per-token scheme and the keys degrade first, corrupting attention scores (which pass through a softmax that amplifies small errors) while the values are still fine. The standard recipe therefore treats them separately: values per-token, keys per-channel, and when budgets are tight it is common to keep keys at a higher bit-width than values — int8 keys with int4 values, say — precisely because a bit spent on keys buys more quality than a bit spent on values.

Decode is memory-bound, so smaller is also faster

The headline benefit is memory, but there is a speed benefit that often surprises people. Token-by-token decoding is memory-bandwidth bound, not compute-bound: at each step the hardware must stream the entire KV cache in to compute attention, and moving those bytes — not the multiply-adds — is the limiting cost.

Halving the cache with int8 roughly halves the bytes read per step, so on a bandwidth-limited device decode can get meaningfully faster as a side effect of quantizing, provided the dequantization is cheap and fused into the attention kernel. The caveat is real: if dequantization is done clumsily — a separate pass that materializes fp16 tensors before attention — you can spend the bandwidth you just saved. The win only lands when the kernel reads integers and dequantizes on the fly.

CPU and small-model implications

On CPU and for small models the case for KV quantization is even stronger, because memory bandwidth is scarcer and there is no large VRAM budget to hide behind. A small model’s weights might fit comfortably in a few gigabytes, but a long chat history in fp16 can quietly become the dominant allocation and the dominant per-token cost. Int8 KV is close to a free win here: the accuracy hit is typically negligible, and halving the streamed cache directly improves the bandwidth-bound decode step that CPUs feel acutely. Int4 KV is where you must be deliberate — group quantization, per-channel keys, often int8 keys with int4 values — but it is what makes very long contexts feasible on a laptop at all. The practical posture: reach for int8 KV almost by default on constrained hardware, and treat int4 as a tunable you validate on your own task rather than switch on blindly.

Pitfalls and a practical checklist

A few mistakes account for most of the disappointing results:

Held together, the discipline is simple: know your bytes-per-token budget, quantize values per-token and keys per-channel, protect the keys with more bits when precision is tight, fuse the dequant, and measure on the workload you actually serve. Do that and int8 KV is nearly free, while int4 becomes a genuine long-context enabler.