KV Cache Sizing for Deployments

Why it matters

KV pool size shapes throughput. Understanding shapes deployment.

The architecture

Compute per-token KV size.

Choose target concurrent tokens.

Reserve model + activation memory.

How it works end to end

Available HBM - model - activations = KV pool.

H100 80GB: ~30GB for KV in Llama 70B.

Trade batch size vs seq length.