KV Cache Quantization Architecture in Depth

Where the memory actually goes

The size of the cache is a product of six terms and nothing else. For a decoder-only transformer:

bytes = 2 * batch * layers * n_kv_heads * head_dim * seq_len * bytes_per_elem
        ^
        K and V

Every term is linear. There is no quadratic anywhere in the cache size - the quadratic you have read about belongs to attention compute over a sequence, not to the state you store. Note also that the head count in the formula is n_kv_heads, not the number of query heads: under grouped-query attention the query heads share a much smaller set of key/value projections, and the cache shrinks by the group factor. The full sizing derivation lives in KV cache math; take the formula as given here.

Put a concrete configuration through it - 80 layers, 8 KV heads, head dimension 128, fp16 storage:

per token = 2 * 80 * 8 * 128 * 2 bytes = 327,680 B = 320 KiB

    8k context,  1 sequence   ->    2.5 GiB
  128k context,  1 sequence   ->   40.0 GiB
    8k context, 32 sequences  ->   80.0 GiB
  128k context,  4 sequences  ->  160.0 GiB

Weights for a model of that shape are a fixed ~140 GB in fp16 and do not move. The cache does. Long context or wide batch and the runtime state overtakes the parameters, and the thing limiting how many users fit on the box stops being the model and starts being the transcript. Halving the bytes per element is the most direct lever you have on that number, because it is the only term in the product that costs you nothing architecturally.

Why a big cache makes decode bandwidth-bound

During autoregressive decode you generate one token per sequence per step. The matmuls are skinny: a batch of 32 sequences is a 32-row activation matrix against weight matrices with thousands of columns. Attention over the cache is worse. For each new query vector you stream every cached key, take a dot product, softmax, then stream every cached value and take a weighted sum. That is roughly two floating-point operations per element loaded.

Two FLOPs per element is an arithmetic intensity far below the ratio any modern accelerator needs to keep its tensor cores busy. The kernel spends its life waiting on HBM. Work the numbers from the example above: 80 GiB of cache at batch 32 and 8k context is about 86 GB, and against roughly 3 TB/s of HBM bandwidth that is close to 29 ms of pure memory traffic per decode step before a single useful FLOP is counted. That is the floor on inter-token latency, and it is set entirely by how many bytes the cache occupies.

This is the whole argument for KV quantization, and it is why the reasoning differs from weight quantization. Storing keys and values in int8 does not make the attention math cheaper - you dequantize back to a compute type before the dot product anyway. It makes the attention math shorter, because half as many bytes cross the memory bus. Bandwidth-bound operations covers the roofline reasoning in general; the KV cache is the canonical example of it.

Keys and values are not the same tensor

The single most important empirical fact about KV quantization is that keys and values have different error structure. Key vectors carry persistent per-channel outliers - certain coordinates of the key dimension run much larger than their neighbours, and the same coordinates are large for every token in the sequence, a structure rotary embedding sharpens rather than averages out. Values show no comparably stable channel structure; their magnitude varies token to token more than coordinate to coordinate. The mechanism is worked through at low bit-widths in outlier channels and group-wise scales, and the general axis question in quantization granularity.

The architectural consequence is what matters here: keys want the channel axis, values want the token axis, and those two axes have very different implications for a cache that is appended to one token at a time. A per-token scale is computed and finalized the instant the vector exists - it never has to be revisited, it lives next to the row it describes, and it composes with an append-only structure with zero bookkeeping. That is why values are easy.

A per-channel key scale cuts across tokens. It describes a column of a tensor whose rows are still arriving. You cannot finalize it when the first token lands, and if you compute it from the tokens seen so far, a later token can exceed it. Everything awkward about KV quantization traces back to that one asymmetry, and every practical scheme is some answer to it - block-local scales, static calibrated ranges, or a full-precision window that buys time. Per-tensor scaling, the coarsest option, sidesteps the bookkeeping entirely and is also the one that gives up the most: a single scale over both the outlier channels and the ordinary ones crushes the ordinary ones into a handful of levels.

INT8, FP8 and INT4 - picking the storage format

INT8 is the safe default. Two-times reduction, 256 uniform levels per scale group, and with per-channel key scales the quality cost is small enough that most workloads will not detect it outside adversarial long-context probes.

FP8 trades levels for dynamic range. The e4m3 encoding gives roughly 4 bits of mantissa but a wide exponent, so it absorbs outliers without needing a fine-grained scale at all - which is precisely why it is the cheapest cache format to implement. On hardware with native FP8 tensor cores the conversion is a hardware instruction rather than an integer multiply-shift, so dequantization cost in the kernel is close to free. Same two-times saving as int8, usually less quality risk on keys, and no scale bookkeeping to fight with. See FP8 formats for the encoding details.

INT4 is where you start negotiating. Sixteen levels per group means the group has to be small for the scale to mean anything, and small groups mean more scales. This is the trap: if you store an fp16 scale and an fp16 zero-point for every group of 32 elements, that is 32 bits of metadata on 128 bits of payload, and your honest bit-width is 5.25, not 4. Quantize the scales themselves, use groups of 64 or 128, or store scales in fp8 - otherwise the format you advertise is not the format you get. Double quantization covers that arithmetic properly, and the int8-versus-int4 accuracy trade is tabulated in bytes per token and int8/int4 trade-offs.

Below 4 bits, KV quantization stops being competitive against structural alternatives - eviction, low-rank projection, sliding windows - covered in KV cache compression.

The architecture: every step explained

Read the diagram as two paths that meet at the kernel.

The write path runs once per token per layer. Immediately after the projection produces a new key and value vector - after rotary embedding is applied, never before, or you would be quantizing a representation that gets rotated afterwards - the runtime computes a scale, rounds to the target format, and writes the packed bytes plus the scale into the cache block. The vector is never stored in full precision, so peak memory equals steady-state memory.

The read path runs once per token per layer per decode step, over the entire history. It loads the packed bytes, loads the matching scales, reconstructs the compute type in registers, and feeds the attention math. The critical property is that the reconstructed tensor is never written back to HBM. If a kernel dequantizes the cache into an fp16 scratch buffer and then calls a standard attention routine on it, you have paid the memory traffic twice and saved nothing - you have made the model slower and less accurate at the same time. Fused dequantization is not an optimization here, it is the entire feature.

The scale layout is what makes or breaks the read path. Key scales indexed per channel are shared across all tokens in a block, so they are a small tensor loaded once per block and reused across the whole dot product. Value scales indexed per token are one scalar per row, loaded alongside the row. Both fit comfortably in shared memory; neither adds meaningful traffic. Get the layout wrong - scales interleaved so that a warp cannot read contiguous payload bytes - and the extra indexing costs more than the volume you saved.

Scales without a calibration pass

For weights the question is how to calibrate. For the KV cache the question is whether you need to at all, and the answer is usually no - calibration-free KV quantization works because the tensor is already in registers when you quantize it. Computing an absolute maximum over 128 elements costs nothing next to the projection that just produced them. Nothing is estimated in advance, nothing drifts when the input distribution shifts, and there is no calibration set to curate or to be wrong about. That is why fp8 and per-token int8 caching are runtime flags while weight quantization is a conversion pipeline.

The serving-side exception is the key tensor under per-channel scaling, for the reason set out above: the scale spans tokens that have not arrived yet. Two answers. Block-local scales compute the per-channel range within a fixed-size block, so the scale is finalized exactly when the block fills and never has to be revised - this is the answer that fits a paged cache, because the block already exists as an allocation unit. Static calibrated ranges collect per-channel maxima offline and work precisely because the outlier channels are stable across inputs, at the cost of reintroducing a calibration step and the drift risk that comes with it. The general trade is in calibration and dynamic vs static quantization; for a serving stack, block-local is usually the simpler answer and the one that needs no offline artifact shipped alongside the model.

The residual window of recent tokens

Keep the most recent tokens in full precision and quantize only the older ones. The numerical case - attention near the current position is sharply peaked, so error in recent entries is weighted far more heavily than error a thousand tokens back - is made in the low-bit treatment. The architectural case is that a window is what makes block-local key scales work at all: tokens land in the fp16 window, and only when a full block's worth has accumulated is that block scaled, packed, and appended to the quantized region.

So size the window to the quantization block, not to a round number. If keys are quantized per channel over blocks of 128 tokens, a 128-token window means the quantizer never touches a partially filled block and never revises a scale. The memory cost is fixed and small - the window does not grow with context - so at 8k context a 128-token window is under two percent of the cache, and at 128k it is a rounding error. This is the highest quality-per-byte knob in the whole design, and it is the one most often left off because it looks like an accuracy hack rather than what it is, a synchronization boundary between the write path and the scaling scheme.

Dequantization is compute you were already going to spend

The instinctive objection to a quantized cache is that you have added work to the inner loop. You have. It does not matter, and understanding why is the point.

Reconstructing a value from an integer and a scale is a convert and a multiply, possibly an add for an asymmetric zero-point - a handful of ALU operations per element, operating on data the kernel had to load regardless. Because attention over the cache runs at an arithmetic intensity of roughly two operations per element, the arithmetic units are idle most of the time waiting on memory; the dequantization slots into that idle time and disappears under the load latency. Meanwhile the load itself got half or a quarter as long. You spent free FLOPs to buy scarce bytes, which is the only trade that matters on a bandwidth-bound kernel.

Three things break the trade. First, unfused dequantization, which round-trips through HBM and inverts the entire benefit. Second, dequantizing on a datapath with no cheap conversion instruction - integer formats need a multiply, fp8 on capable hardware needs neither, and on hardware without fp8 support the emulation can cost more than it saves. Third, layouts that force uncoalesced loads: if the packed cache and its scales are not laid out so a warp reads contiguous bytes, you lose more to poor access patterns than you gained in volume. The same IO-aware discipline that makes FlashAttention work applies unchanged - and in practice the quantized cache kernel is a FlashAttention variant with an unpack step in the load.

How it composes with paging, GQA and offload

KV quantization is multiplicative with the other memory levers, not redundant with them, but the interactions are worth naming.

Paged attention. The page is the natural quantization group - it is already a fixed-size, contiguous chunk of the cache with its own metadata, so scales attach to the block descriptor and the two schemes fit together with no friction. Two caveats. A block size chosen for fp16 holds the same number of tokens after quantization but occupies fewer bytes, so your fragmentation and allocator tuning both shift. And copy-on-write sharing of a block between sequences now shares a scale as well, which is fine as long as scales are block-local and immutable once sealed. See paged KV cache for the block-table mechanics.

GQA and MQA already shrank the cache by the group factor, and this is the interaction people get wrong. A model with 64 query heads and 8 KV heads has one-eighth the cache of the multi-head version, so int8 on top gives sixteen-times over the naive baseline - but the remaining cache is smaller in absolute terms, so the fraction of your memory budget you can win back by quantizing it is correspondingly smaller. On an MQA model with a short context the cache may simply not be the constraint any more. Details in GQA math.

Offload. Quantization and tiering to host memory attack the same problem from opposite ends, and quantizing first makes offload strictly cheaper by shrinking what crosses PCIe - a link roughly an order of magnitude slower than HBM, where halving the bytes buys far more than it does on-device.

Evaluating the damage: perplexity will lie to you

Perplexity is close to useless as a KV quantization gate, and relying on it is the most common way teams ship a regression they did not intend.

The reason is structural. Perplexity averages next-token loss over a corpus, and most next tokens are predictable from local context. Local context is exactly the part of the cache that is freshest, least quantized under a residual window, and most heavily attended. Damage from a quantized cache concentrates in the opposite regime: the long, flat tail of attention over distant tokens, where a small number of entries must win a softmax against thousands of near-ties. Add quantization noise and the argmax over that tail flips. Averaged over a corpus the effect on mean loss is a rounding error. On a task that depends on retrieving one specific distant fact, it is a wrong answer.

So test on retrieval-shaped long-context work: needle-in-a-haystack probes at your actual maximum context, multi-hop questions over long documents, exact reproduction of an identifier or quotation from early in the transcript. Sweep the context length rather than testing a single point - degradation is characteristically flat and then sharply not, and a gate that only checks 4k will pass a configuration that fails at 64k. Evaluate the full serving configuration too, since sampling temperature and speculative decoding both interact with the noise you just introduced: a draft model verified against a noisier target rejects more, and the acceptance-rate drop can eat the memory win.

When not to bother

Reach for a quantized cache when the cache is what is actually limiting you, and not otherwise.

Skip it when contexts are short and batches are small: at 2k tokens with a handful of concurrent requests the cache is a small fraction of a machine already holding the weights, and you have traded accuracy for headroom you were not using. Skip it on an MQA model with modest context for the same reason - the group factor did the work already. Skip it if your bottleneck is prefill rather than decode, because prefill is compute-bound and a smaller cache does not help a bottleneck that is not memory traffic.

Skip it, too, if the runtime you are on lacks a fused dequantizing attention kernel for your chosen format and hardware. An unfused implementation is strictly worse than fp16 on both axes at once. Check that the kernel exists before you plan capacity around the savings, and benchmark decode throughput rather than trusting the format's nominal ratio.

And when your request pattern shares long prefixes across users - a system prompt, a document everyone is asking about - evaluate prefix caching first. Deduplicating identical prefixes can beat a two-times format saving outright and costs nothing in accuracy. The two compose, so the question is which to spend engineering time on, not which to pick.