Why it matters

Continuous batching dominates LLM serving. Understanding shapes design.

Advertisement

The architecture

Per-token step: any request can join or finish.

Scheduler adds new requests each step.

Continuous batchingRunning batchcurrent tokensPer-token stepgenerate 1 tokenAdd / finishdynamic membershipaka in-flight batching or iteration-level scheduling; vLLM + TGI + TRT-LLM support
Continuous batch.
Advertisement

How it works end to end

Orca paper 2022.

Per-token join + evict.

Max batch tokens across sequences.

PagedAttention supports.