A serving system does not have ‘a latency.’ It has two latency SLOs with different drivers, and a queue in front of both. Time-to-first-token is a queueing problem; time-per-output-token is a contention problem; and the two fight each other, because the work that produces one request’s first token is the work that stalls everyone else’s next token. This piece does the arithmetic: how the two SLOs decompose, why a long prefill blows up other requests’ TPOT and how big a chunk you may safely admit, why the tail explodes near saturation and why the textbook M/M/1 answer is optimistic for LLM traffic, what admission control actually buys, and how to size a fleet for a p99 target rather than for an average.
Two SLOs, not one
Interactive LLM serving is judged on two clocks. TTFT (time-to-first-token) is how long the user stares at nothing. TPOT (time-per-output-token, sometimes ITL) is how fast text then streams. End-to-end latency is just their composition:
E2E = TTFT + (N_out − 1) · TPOT
TTFT = t_queue + t_prefill
t_prefill ≈ P_tokens / prefill_rateEvery number below comes from one locked parameter set: prefill runs at 8,000 tok/s per replica, a decode step takes 30 ms at batch B = 32, a typical prompt is 2,000 tokens (long tail 4,000), output is 300 tokens, and the SLOs are TTFT p99 ≤ 1,000 ms and TPOT p99 ≤ 50 ms. An unqueued typical request gives t_prefill = 2000/8000 = 250 ms and E2E = 250 + 299 × 30 = 9,220 ms. Note the shape: TTFT is 3% of the total, yet it is the number users complain about.
What actually drives each number
The two SLOs load different parts of the machine. Prefill is a big batched GEMM over P tokens at once — compute-bound, and quadratic in sequence length through attention (derived in the sequence-length and roofline articles; here we just consume it as prefill_rate). Decode is one token per sequence per step, re-reading the whole weight matrix to do a matvec: memory-bandwidth-bound, and nearly flat in batch size until the batch is large enough to matter.
The scheduling consequence is what belongs here. TTFT absorbs queue wait — time spent admitted-but-not-started — so it responds to arrival rate and to how long the server is busy with other work. TPOT absorbs contention inside the running batch — it degrades because of who you are running alongside, not because of who is waiting behind. That asymmetry is why shedding load fixes TTFT immediately but only helps TPOT to the extent that it caps concurrency.
Prefill/decode interference, in numbers
Here is the interference that makes naive schedulers fail. Thirty-two sequences are decoding happily at 30 ms per step. A new request arrives with a 4,000-token prompt. If the scheduler runs that prefill as one iteration, that iteration costs 4000/8000 = 500 ms of pure prefill.
normal step = 30 ms → TPOT ok
prefill step = 30 + 500 = 530 ms → 10.6× over the 50 ms SLO
victims = all 32 in-flight sequencesOne request’s TTFT win costs thirty-two requests a 530 ms stutter, and it hides from the mean: averaged over 300 steps it adds only 500/300 ≈ 1.7 ms. It lands squarely on p99, which is the number in the SLO. A rare, large stall is invisible to the mean and fatal to the tail.
Chunked prefill: sizing the chunk from the TPOT budget
The fix is to stop treating prefill as atomic. Chunked prefill splits the prompt into pieces of C tokens and runs one piece per iteration, piggybacking the decode tokens onto the same forward pass. The chunk size falls straight out of the TPOT budget:
C_max = (TPOT_slo − t_decode) × prefill_rate
= (50 − 30) ms × 8,000 tok/s = 160 tokensTreating the chunk and the piggybacked decode as additive is a conservative upper bound — once a chunk saturates compute, the decode tokens ride along in the same weight read almost free — so 160 is a safe floor, not the physics. Now the tension: a 4,000-token prompt needs 25 chunks, and 25 × 50 ms = 1,250 ms of TTFT — over the 1,000 ms budget. Chunking does not delete the conflict; it converts a TPOT violation into a TTFT one. You resolve it by relaxing one SLO, capping prompt length at admission, or disaggregating prefill onto separate replicas.
Utilization and the tail: the honest queueing picture
Queue wait is governed by utilization ρ = λ / μ. For the textbook M/M/1 server, mean sojourn time is W = S / (1 − ρ), and sojourn time is exponentially distributed, so the p99 is a fixed multiple of the mean: p99 = ln(100) · W ≈ 4.6 W.
| ρ | mean W (× S) | p99 (× S) |
|---|---|---|
| 0.50 | 2.0 | 9.2 |
| 0.70 | 3.3 | 15.4 |
| 0.90 | 10.0 | 46.1 |
| 0.95 | 20.0 | 92.1 |
Read the ratio correctly. The 4.6× gap is constant in ρ; what explodes is the mean it multiplies, because 1/(1−ρ) is a hyperbola. Going from 90% to 95% utilization buys 5% more traffic and doubles the tail. That is why capacity plans for latency-sensitive serving are written in headroom, not in efficiency.
Why M/M/1 flatters LLM traffic
M/M/1 assumes exponential service times. LLM service time is dominated by output length, which is emphatically not exponential: most answers are short, a few run to the token cap, and the distribution is heavy-tailed. The Pollaczek–Khinchine formula prices that variability:
W_q = ρ · S · (1 + C_v^2) / (2 (1 − ρ))
C_v = σ_S / S (C_v = 1 recovers M/M/1)If output lengths give C_v = 2 — entirely ordinary when a 512-token cap sits next to a 40-token mean — then (1 + 4)/2 = 2.5, and the waiting component of every row above — the part of W beyond the one service time S — runs 2.5× larger at the same utilization. The tail is worse than that rescale suggests: with heavy-tailed service, sojourn time stops being exponential, so the tidy 4.6× multiplier becomes a floor, not a fixed ratio. The practical reading: the knee moves left. A fleet you would have run at ρ = 0.8 under exponential assumptions needs to sit near 0.6, and cutting variance (token caps, length-bucketed queues) is as valuable as adding hardware.
Policy choice: which objective are you optimizing?
FCFS is fair and predictable but lets one 4,000-token prompt delay everything behind it. Shortest-job-first and SRPT provably minimize mean latency — and that is precisely the trap, because an SLO does not ask for a small mean. It asks for a bounded miss rate. Mean-optimal is not deadline-optimal: SRPT will happily starve the longest 1% of requests to shave milliseconds off the other 99%, which is a p99 catastrophe.
Worse, SJF needs the job length, and for generation the length is unknown until the model emits its stop token. You must predict it, and prediction error converts directly into starvation: a request mispredicted as long is deprioritized, ages, and misses. Earliest-deadline-first is the deadline-aware answer, but note its real guarantee — EDF is optimal only when the set is feasible. Under overload it exhibits the domino effect, spending capacity on the request nearest its deadline, which is often the one already doomed.
Admission control and load shedding
That last point is the bridge. EDF, priorities, and clever ordering are all conservation laws: they move latency around, they do not create capacity. Once ρ → 1, no policy meets the SLO, so the only lever that keeps a deadline scheduler meaningful is admission control — refusing work so the admitted set stays feasible.
The queue-depth cap follows from the SLO arithmetic directly. If the wait budget is TTFT_slo − t_prefill = 1000 − 250 = 750 ms and a replica drains admitted requests at 3.5 req/s, anything past 0.75 × 3.5 ≈ 2.6 queued requests is already guaranteed to miss: admitting it cannot help that user and does hurt everyone ahead. So reject fast with a 429, shed the lowest tier, or reject on predicted prompt length before prefill starts. A fast failure is a better product than a slow success.
Goodput: the only metric the SLO respects
Throughput counts tokens produced. Goodput counts tokens produced within the SLO, and only the second one is worth optimizing. Take the decode batch-size curve for our replica, where TPOT bends upward as the batch stops being bandwidth-bound (one weight read amortized over more tokens) and starts being compute-bound:
B=16 TPOT 20 ms → 16/0.020 = 800 tok/s feasible
B=32 TPOT 30 ms → 32/0.030 = 1067 tok/s feasible
B=64 TPOT 55 ms → 64/0.055 = 1164 tok/s VIOLATES 50 msRaw throughput peaks at B = 64. Goodput there is zero — every token is late, so none of them count. The SLO-feasible optimum is B = 32. This is why a serving benchmark quoting peak tokens/s without a latency constraint is not measuring anything a user experiences, and why max batch size is an SLO parameter rather than a throughput knob.
Worked example: sizing a fleet for p99
Target λ = 30 req/s at the SLOs above. Start with Little’s law, which converts a rate into a concurrency requirement. Each request holds a batch slot for its whole life:
T_hold = 0.25 + 300 × 0.030 = 9.25 s
a = λ · T_hold = 30 × 9.25 = 277.5 concurrent slots
c = a + β√a = 277.5 + 2 × 16.7 ≈ 311 slotsThe square-root staffing term is the tail headroom: β = 2 keeps the probability of queueing small — the many-server version of the headroom the utilization table demanded. At 32 slots per replica that is 311/32 = 9.7 → 10 replicas, plus one so a single failure does not push the survivors past the knee: 11. State the assumption out loud: 30 ms was measured at B = 32, so T_hold is already the at-capacity value, which makes this sizing conservative rather than optimistic. Size on λ alone and you order 9 and miss every p99.
(TPOT_slo − t_decode) × prefill_rate, accepting that you have converted a TPOT problem into a TTFT one. Queue wait grows as 1/(1−ρ) and heavy-tailed output lengths multiply it by (1 + C_v^2)/2, so the safe operating point sits lower than textbook queueing suggests. No policy creates capacity — EDF is optimal only on a feasible set, which is what makes admission control load-bearing rather than optional. Optimize goodput, not throughput, and size fleets with Little’s law plus explicit headroom, never on the average.