Most writing about LLM serving zooms in on one clever mechanism — paged attention, continuous batching, speculative decoding — and leaves you without a picture of the machine those mechanisms live inside. This article is the opposite: a map. We follow one request from the load balancer to the moment its last token lands in the client’s buffer, naming each stage and pointing at the article that develops it, then account for the latency budget, catalogue how each stage fails, and end at the cluster level.
The map: one request, eight stages
A serving stack is easiest to hold in your head as a pipeline with a loop in the middle. A request is admitted, queued, then picked up by a scheduler that assembles a batch. The executor runs one forward pass over that batch, spread across however many GPUs the weights need. The KV cache manager hands out memory for the attention state it produces. A sampling stage turns the final logits into token IDs, a detokenizer turns those into text, and a streaming layer pushes the text back. Then the loop repeats, once per generated token.
Everything below is a variation on that skeleton, and the interesting engineering sits at the loop’s edges: who gets into the batch, what happens when the cache runs out, and what the client sees while it waits. The figure shows the same system as deployed components rather than request stages — note that everything from the scheduler onward lives inside its single serving-engine box.
Ingress and admission: the front door
Traffic enters through an L7 proxy that terminates TLS and parses HTTP/2 or gRPC, then usually a mesh enforcing mTLS and one shared retry policy. That last point matters more than it sounds: without it, one client team’s aggressive retries multiply load on an already-saturated backend, and GPU backends saturate in seconds.
The GPU-specific concern is admission. Unlike a stateless web service, an LLM engine cannot absorb an unbounded queue: every accepted request reserves KV cache memory for its whole lifetime, and a request that waits four minutes is one the user already abandoned. So the front door does real work — count tokens, reject prompts over the context limit before they reach a GPU, apply per-tenant concurrency limits, and shed load with a fast 429 rather than a slow timeout. Fast rejection is a feature.
The scheduler: what runs in this step
The scheduler is the heart of the engine. Each iteration it weighs the waiting queue against the running sequences and decides what the next forward pass covers, subject to two hard budgets: a token budget (how much prefill compute one step absorbs) and a memory budget (how many KV blocks are free). It runs at iteration granularity, not request granularity — a finished sequence leaves the batch immediately and a waiting one takes its slot, rather than the whole batch waiting on its slowest member.
That is continuous batching, which has its own article; the point here is where it sits. The scheduler is the only component that sees queueing delay and cache pressure at once, making it the natural home for priority classes, for chunking a long prefill, and for preempting a sequence when memory runs short. Every fairness question lands here.
The model executor and the parallelism layout
Below the scheduler is the executor: worker processes, one per GPU, holding the weights and running the forward pass. How those weights are split is the layout decision that shapes everything else.
Tensor parallelism shards individual weight matrices across GPUs, so every layer ends in an all-reduce recombining partial results. That collective fires twice per transformer block and sits on the critical path of every token, which is why TP groups stay inside one NVLink/NVSwitch domain — over PCIe or Ethernet it would dominate step time. Pipeline parallelism instead assigns layer ranges to different GPUs and passes activations between them; it tolerates slower links but introduces bubbles, and has its own article. Rule of thumb: fill one node with tensor parallelism first, reach for pipeline parallelism only when the model will not fit. Weights load from an object store at pod start — roughly 140 GB for a 70B model in FP16 — so cold start takes minutes.
The KV cache manager: the real capacity constraint
Once the weights are resident, whatever is left in HBM becomes the KV cache pool — and that pool, not FLOPs, usually caps how many concurrent users a pod holds. Each sequence accumulates a key and value vector per layer per token, so its footprint grows linearly with generated length, and you do not know the final size when you admit it.
Engines solve this by allocating fixed-size blocks tracked by a per-sequence block table, so a sequence grows a block at a time rather than reserving its maximum up front, and identical prefixes can share physical blocks. Those mechanics belong to the paged-KV-cache article. What the map needs you to see is the coupling: the cache manager reports free-block count to the scheduler every iteration, the scheduler admits or preempts on that number, and a routing layer that ignores prefix locality destroys the reuse this component exists to provide.
Sampling and detokenization: the end of every step
The forward pass ends with a logit vector per sequence over the whole vocabulary — commonly 100k or more entries in current models. At batch size 256 that is tens of millions of elements produced every token, which sampling must reduce: apply temperature, penalties, and any logit bias or grammar mask, then take top-k and top-p and draw one ID per sequence.
People assume this is free because it is not a matmul. A naive top-p sorts the vocabulary per sequence; constrained decoding builds a mask over it per step. Both are real kernels whose cost grows with batch size, exactly where batching was supposed to pay off. Detokenization is the sharper trap: it runs on the CPU and is stateful, since tokens map to partial UTF-8 sequences. Done synchronously in the thread that launches the next step, it idles the GPU; well-built engines push it to a separate process.
Streaming the response back
The loop’s user-visible product is a stream, usually server-sent events or gRPC, carrying a token at a time. That path must stay clear end to end: any buffering proxy between engine and client — a gateway waiting for a full response body, compression filling a buffer before it flushes — turns a well-tuned streaming engine into one that appears to hang and then dump. It is also why timeouts along the chain must exceed the longest plausible generation, not the median.
The reverse direction matters as much and is routinely forgotten: cancellation. When a user closes a tab, the disconnect must propagate through proxy and mesh to the scheduler so the sequence is dropped and its KV blocks freed. Otherwise the pod keeps generating tokens for nobody while holding cache live requests need — and in a bursty agentic workload full of abandoned calls, that gap alone can eat a large slice of capacity.
Where the latency budget goes
Serving latency has two halves with different physics, and conflating them is the most common analysis mistake. Time to first token covers admission, queueing, and prefill. Prefill processes the whole prompt at once, is compute-bound, and scales with prompt length, so TTFT is queueing delay plus a term proportional to input tokens. Inter-token latency covers the rest: one forward pass per token, reading all the weights and the whole KV cache from HBM to produce a single token per sequence. That is memory-bandwidth-bound, which is why decode wants a large batch — more sequences amortize one weight read.
Because the phases want opposite things, mixing them in one step lets a long prefill stall everyone’s decode, surfacing as ITL jitter. The standard responses are chunking prefill, or splitting the phases onto separate GPU pools — prefill/decode disaggregation, which has its own article. Speculative decoding attacks the other side, trading spare compute for fewer sequential steps.
How the pieces fail
Each stage has a characteristic failure, and they look nothing alike from outside. Head-of-line blocking: one 100k-token prompt takes the prefill budget and every short request behind it inherits its latency. Cache exhaustion: free blocks hit zero, the scheduler preempts, and preempted work is swapped out or recomputed — either way throughput drops non-linearly at exactly the load level where you needed it.
Straggler in a collective: a tensor-parallel group is a barrier, so one GPU throttling on temperature or hitting ECC retries slows all eight, and the symptom is uniform slowness with no obvious culprit. Wedged worker: the nastiest — a hung NCCL collective leaves the HTTP server responsive, so a shallow liveness probe passes while the pod serves nothing, which is why readiness must exercise a real forward pass. And cold start: a pod that dies during a spike cannot be replaced within it.
Capacity planning at the cluster level
Zoom out and the unit of capacity is not a GPU but a replica: one parallel group holding a full copy of the model. You scale by adding replicas, so a model needing eight GPUs scales in eight-GPU steps — coarse granularity that makes the shape of your bursts a first-class design input.
Route across replicas with more than round-robin. Least-outstanding-tokens beats least-connections, because requests differ by orders of magnitude in cost, and prefix-affinity routing — sending requests that share a system prompt to the same replica — is what makes prefix caching pay. Do not autoscale on GPU utilization: during decode the GPU is busy at nearly any load, so the metric is pegged and tells you nothing. Scale on queue depth, waiting tokens, or measured TTFT against your SLO. Then accept what multi-minute cold starts imply: you cannot chase a burst reactively. Bursts are absorbed by headroom — warm replicas you pay for and do not use — and the planning question is how much.