Chunked prefill is a serving trick that answers one awkward question: what happens to everyone else’s tokens while the server chews through your 8,000-token prompt? In a naive scheduler the answer is ‘they wait’ — a single long prefill monopolizes the batch and every user already streaming a reply sees a visible stall. Chunked prefill fixes this by refusing to run a whole prompt in one shot. It splits the prompt into fixed-size chunks, processes one chunk per iteration, and uses the spare room in that iteration to piggyback the ongoing decode steps of other requests. The result is a batch whose per-step cost stays roughly constant, so inter-token latency stops spiking. This piece works through why the stall happens, the math of chunking, the compute-versus-latency tradeoff that sets the chunk size, and the token-budget scheduling that Sarathi-Serve popularized and vLLM now ships by default.
Two phases, one shared batch
An LLM server runs two very different workloads on the same hardware. Prefill ingests a whole prompt of L tokens in one dense, parallel forward pass — a big, compute-bound GEMM over [L, d]. Decode generates one token at a time, a skinny [1, d] matrix-vector product per running request, and is memory-bandwidth-bound. A modern server batches many requests together to keep the device busy, so at any instant the batch contains a mix: some sequences are still decoding their reply, and occasionally a new request arrives that must be prefilled.
The trouble is that these two jobs fight over the same iteration. A decode step for a batch of B requests touches B new tokens. A prefill of a long prompt touches L tokens — possibly thousands at once. When the scheduler tries to run that prefill as a single unit, it produces one enormous iteration whose duration is set by L, not by B. Everything else in the batch is held hostage until it finishes.
The stall: head-of-line blocking
Concretely, imagine ten users mid-reply, each expecting a fresh token roughly every 30 ms — a smooth inter-token latency (ITL). Now an eleventh user pastes a 6,000-token document. A prefill-first scheduler runs that prefill in one iteration that might take 300 ms. For that whole window the ten streaming users get nothing: their next token is stuck behind the giant prefill. This is classic head-of-line blocking, and it shows up as an ugly ITL spike — the stream freezes, then lurches.
You cannot fix this by simply prioritizing decodes and deferring the prefill either: then the new user’s time-to-first-token (TTFT) balloons while they wait for a gap in decoding. The two objectives — low TTFT for arrivals and low, steady ITL for streams — are in direct tension whenever a prefill is large enough to dominate an iteration. Chunking is what lets you serve both at once instead of trading one for the other.
The core idea: split the prompt
Chunked prefill breaks the atomic assumption. Instead of processing all L prompt tokens in one forward pass, it slices the prompt into chunks of size C and feeds one chunk per iteration:
n_chunks = ceil(L / C)
chunk 1: tokens [0 .. C) -> partial forward pass
chunk 2: tokens [C .. 2C) -> partial forward pass
...
chunk k: tokens [(k-1)C .. kC) -> the new tokens attend to
ALL previous tokens via KV cacheEach chunk runs the full stack of layers, but only over its C new positions. Crucially, the keys and values of earlier chunks are already sitting in the KV cache, so chunk k’s tokens still attend to the entire prefix [0, kC) — there is no loss of context or accuracy. The first token is emitted only after the last chunk completes; chunking changes the schedule of the prefill work, not its result.
Piggybacking decode onto each chunk
Splitting the prompt is only half the trick. The real payoff comes from what you do with the leftover capacity of each iteration. A prefill chunk of C tokens is small; the batch can afford to process more tokens in the same step. So the scheduler co-schedules the chunk together with the decode steps of every other running request — each contributes its one new token — forming a single fused batch:
iteration tokens = C (one prefill chunk)
+ D (one decode token per running request)This is the Sarathi-Serve insight (Agrawal et al., 2023): because decode is memory-bound and prefill is compute-bound, they use different parts of the hardware, and fusing them into one iteration is nearly free. The decodes ride along on a pass that had to load the weights anyway, raising the arithmetic intensity of an otherwise wasteful decode-only step. Nobody stalls: the ten streaming users keep getting a token every iteration, and the long prompt advances one chunk at a time in the background.
The token budget that keeps iterations uniform
The scheduler’s knob is a fixed per-iteration token budget B — the maximum number of tokens any single iteration may process. Each step first admits the mandatory decode tokens (one per running request, D of them), then fills the remaining room with a prefill chunk:
C = B - D (chunk size adapts to the decode load)
every iteration processes ~B tokens
=> every iteration costs ~the same wall-clock time
=> ITL is smooth and predictableBecause the total token count per step is clamped near B, the iteration time barely varies whether or not a prefill is in flight. When many requests are decoding, D is large and the prefill chunk shrinks to fit; when the batch is quiet, a bigger chunk slides in and prefill races ahead. This adaptive C = B - D is what converts a bursty, spiky workload into a stream of near-identical iterations — the whole point of the exercise.
Chunking barely changes the FLOP count
A natural worry: does slicing the prompt cost more compute? Almost none. The dense linear layers cost 2 · N_params FLOPs per token no matter how the tokens are grouped, so summed over all chunks the projection cost is still ≈ 2 · N_params · L — identical to an unchunked prefill. The attention term is subtler but also essentially preserved. Chunk k has C query tokens attending to about kC cached keys, costing ~C · kC score operations; summing k = 1 … n gives:
Σ_k C · kC = C^2 · n(n+1)/2 ≈ L^2 / 2— the same O(L^2) triangle a single-pass prefill computes. The genuine overheads are second-order: smaller GEMMs have slightly lower arithmetic intensity than one giant matmul, and each chunk must re-read the growing KV cache from memory. Chunking is a scheduling technique, not a FLOP-saving one — it trades a few points of raw prefill throughput for a dramatically better latency profile.
The compute-versus-latency tradeoff of chunk size
Chunk size C (equivalently the budget B) is the central dial, and it pulls in two directions. Make C small and every iteration is short, so decodes interleave tightly and ITL is silky — but the prompt now takes ceil(L/C) iterations, the GEMMs are skinny, and the KV cache is re-read many times, so prefill efficiency (and TTFT) degrades. Make C large and prefill runs at near-peak efficiency with excellent TTFT — but a big chunk reintroduces exactly the long, decode-blocking iteration you were trying to avoid.
The art is to pick the largest chunk whose iteration still fits inside your ITL budget. If users tolerate a 40 ms token cadence and one chunk-plus-decode iteration at C = 512 lands around 35 ms, you are comfortably inside budget while keeping prefill efficient. Push to C = 2048 and the iteration might blow past 40 ms, stuttering the streams. There is no universal number: it depends on model size, hardware speed, and how many decodes typically share the batch.
A worked example
Take a prompt of L = 8192 tokens, a token budget B = 512, and a batch with D = 32 requests currently decoding. Each iteration must carry the 32 decode tokens, leaving a prefill chunk of C = B - D = 480:
chunks needed = ceil(8192 / 480) = 18 iterations
per-iteration = 480 prefill + 32 decode = 512 tokens (= B)
if one iteration ≈ 30 ms:
new user's TTFT ≈ 18 × 30 ms = 540 ms
streaming users get 1 token every ~30 ms throughoutCompare the unchunked path: the 8,192-token prefill runs as one iteration of perhaps 480 ms, during which all 32 streaming users freeze — a single 480 ms ITL spike. Chunking spreads that same prefill work across 18 modest steps. The new user waits a little longer for their first token (540 ms versus 480 ms — a small TTFT tax), and in exchange nobody suffers the half-second stall. That trade — slightly higher TTFT for a flat ITL — is the deal chunked prefill offers.
CPU-SLM implications and common pitfalls
On CPU-served small language models the effect is amplified. CPUs have far less raw FLOP headroom than GPUs, so a long single-pass prefill occupies the cores for a painfully long stretch and the stall is even more visible. Bounding each iteration with a modest token budget keeps an interactive SLM responsive on commodity hardware, and the decode piggybacking is especially valuable because CPU decode steps are memory-bound and otherwise leave the vector units idle — a fused prefill chunk soaks up that slack.
Two pitfalls recur. First, a chunk size chosen for throughput alone will quietly reintroduce ITL spikes; always validate C against your actual latency target, not just tokens-per-second. Second, remember that chunking does not reduce total work — if TTFT on very long prompts is your bottleneck, chunking will nudge it slightly worse, and the real fix lives elsewhere (prefix caching, faster attention kernels, or more compute). Used for what it is — a latency-smoothing scheduler — chunked prefill is one of the highest-leverage, lowest-cost wins in modern LLM serving.
Related: Chunked Prefill in Serving covers sizing the token budget against a TPOT SLO and the throughput tax it buys.