Why it matters

KV cache math shapes serving. Understanding shapes capacity planning.

Advertisement

The architecture

Per token per layer: 2 (K + V) * heads * dim.

Multiply by seq length + batch.

KV cache formulaPer token2 * heads * dimPer sequence* seq lengthBatch total* batch sizeLlama 3 8B: ~0.5MB per 1k tokens; GQA reduces by heads ratio
KV cache.
Advertisement

How it works end to end

Formula: 2 * layers * batch * seq * heads * head_dim * bytes.

bf16: 2 bytes.

int8: 1 byte via KV quant.

GQA reduces by heads / kv_heads.