Why it matters
Chunked prefill improves latency mix. Understanding shapes serving.
Advertisement
The architecture
Chunk size C tokens per step.
Mix chunk prefill with decode tokens.
Advertisement
How it works end to end
Chunk size tunable.
Piggyback decodes on chunks.
Better ITL for mixed loads.