Two SLOs, not one
Interactive LLM serving is judged on two clocks. TTFT (time-to-first-token) is how long the user stares at nothing. TPOT (time-per-output-token, sometimes ITL) is how fast text then streams. End-to-end latency is just their composition:
E2E = TTFT + (N_out − 1) · TPOT
TTFT = t_queue + t_prefill
t_prefill ≈ P_tokens / prefill_rateEvery number below comes from one locked parameter set: prefill runs at 8,000 tok/s per replica, a decode step takes 30 ms at batch B = 32, a typical prompt is 2,000 tokens (long tail 4,000), output is 300 tokens, and the SLOs are TTFT p99 ≤ 1,000 ms and TPOT p99 ≤ 50 ms. An unqueued typical request gives t_prefill = 2000/8000 = 250 ms and E2E = 250 + 299 × 30 = 9,220 ms. Note the shape: TTFT is 3% of the total, yet it is the number users complain about.
What actually drives each number
The two SLOs load different parts of the machine. Prefill is a big batched GEMM over P tokens at once — compute-bound, and quadratic in sequence length through attention (derived in the sequence-length and roofline articles; here we just consume it as prefill_rate). Decode is one token per sequence per step, re-reading the whole weight matrix to do a matvec: memory-bandwidth-bound, and nearly flat in batch size until the batch is large enough to matter.
The scheduling consequence is what belongs here. TTFT absorbs queue wait — time spent admitted-but-not-started — so it responds to arrival rate and to how long the server is busy with other work. TPOT absorbs contention inside the running batch — it degrades because of who you are running alongside, not because of who is waiting behind. That asymmetry is why shedding load fixes TTFT immediately but only helps TPOT to the extent that it caps concurrency.
Prefill/decode interference, in numbers
Here is the interference that makes naive schedulers fail. Thirty-two sequences are decoding happily at 30 ms per step. A new request arrives with a 4,000-token prompt. If the scheduler runs that prefill as one iteration, that iteration costs 4000/8000 = 500 ms of pure prefill.
normal step = 30 ms → TPOT ok
prefill step = 30 + 500 = 530 ms → 10.6× over the 50 ms SLO
victims = all 32 in-flight sequencesOne request’s TTFT win costs thirty-two requests a 530 ms stutter, and it hides from the mean: averaged over 300 steps it adds only 500/300 ≈ 1.7 ms. It lands squarely on p99, which is the number in the SLO. A rare, large stall is invisible to the mean and fatal to the tail.
Chunked prefill: sizing the chunk from the TPOT budget
The fix is to stop treating prefill as atomic. Chunked prefill splits the prompt into pieces of C tokens and runs one piece per iteration, piggybacking the decode tokens onto the same forward pass. The chunk size falls straight out of the TPOT budget:
C_max = (TPOT_slo − t_decode) × prefill_rate
= (50 − 30) ms × 8,000 tok/s = 160 tokensTreating the chunk and the piggybacked decode as additive is a conservative upper bound — once a chunk saturates compute, the decode tokens ride along in the same weight read almost free — so 160 is a safe floor, not the physics. Now the tension: a 4,000-token prompt needs 25 chunks, and 25 × 50 ms = 1,250 ms of TTFT — over the 1,000 ms budget. Chunking does not delete the conflict; it converts a TPOT violation into a TTFT one. You resolve it by relaxing one SLO, capping prompt length at admission, or disaggregating prefill onto separate replicas.
Utilization and the tail: the honest queueing picture
Queue wait is governed by utilization ρ = λ / μ. For the textbook M/M/1 server, mean sojourn time is W = S / (1 − ρ), and sojourn time is exponentially distributed, so the p99 is a fixed multiple of the mean: p99 = ln(100) · W ≈ 4.6 W.
| ρ | mean W (× S) | p99 (× S) |
|---|---|---|
| 0.50 | 2.0 | 9.2 |
| 0.70 | 3.3 | 15.4 |
| 0.90 | 10.0 | 46.1 |
| 0.95 | 20.0 | 92.1 |
Read the ratio correctly. The 4.6× gap is constant in ρ; what explodes is the mean it multiplies, because 1/(1−ρ) is a hyperbola. Going from 90% to 95% utilization buys 5% more traffic and doubles the tail. That is why capacity plans for latency-sensitive serving are written in headroom, not in efficiency.
Why M/M/1 flatters LLM traffic
M/M/1 assumes exponential service times. LLM service time is dominated by output length, which is emphatically not exponential: most answers are short, a few run to the token cap, and the distribution is heavy-tailed. The Pollaczek–Khinchine formula prices that variability:
W_q = ρ · S · (1 + C_v^2) / (2 (1 − ρ))
C_v = σ_S / S (C_v = 1 recovers M/M/1)If output lengths give C_v = 2 — entirely ordinary when a 512-token cap sits next to a 40-token mean — then (1 + 4)/2 = 2.5, and the waiting component of every row above — the part of W beyond the one service time S — runs 2.5× larger at the same utilization. The tail is worse than that rescale suggests: with heavy-tailed service, sojourn time stops being exponential, so the tidy 4.6× multiplier becomes a floor, not a fixed ratio. The practical reading: the knee moves left. A fleet you would have run at ρ = 0.8 under exponential assumptions needs to sit near 0.6, and cutting variance (token caps, length-bucketed queues) is as valuable as adding hardware.
Policy choice: which objective are you optimizing?
FCFS is fair and predictable but lets one 4,000-token prompt delay everything behind it. Shortest-job-first and SRPT provably minimize mean latency — and that is precisely the trap, because an SLO does not ask for a small mean. It asks for a bounded miss rate. Mean-optimal is not deadline-optimal: SRPT will happily starve the longest 1% of requests to shave milliseconds off the other 99%, which is a p99 catastrophe.
Worse, SJF needs the job length, and for generation the length is unknown until the model emits its stop token. You must predict it, and prediction error converts directly into starvation: a request mispredicted as long is deprioritized, ages, and misses. Earliest-deadline-first is the deadline-aware answer, but note its real guarantee — EDF is optimal only when the set is feasible. Under overload it exhibits the domino effect, spending capacity on the request nearest its deadline, which is often the one already doomed.