Why it matters
Throughput math shapes ops economics. Understanding shapes design.
Advertisement
The architecture
Per GPU: HBM_BW / bytes_per_token / bytes_per_param.
Batch amortizes.
Advertisement
How it works end to end
Continuous batching scales throughput.
KV cache limits practical batch.
Speculative decoding boosts single-stream.