Why it matters

LLM math 101 shapes ops. Understanding shapes serving design.

Advertisement

The architecture

Prefill: compute-bound.

Decode: memory-bound.

KV cache: memory consumer.

LLM serving math trianglePrefillcompute-boundDecodememory-boundKV cachememory poolThree axes trade against each other; SLOs pick point on triangle
LLM math 101.
Advertisement

How it works end to end

Trade-offs: batch vs latency, quantize vs quality, context vs throughput.

Match config to SLO.