Most articles in this series zoom into one number — the cost of an attention step, the size of a KV cache, the throughput of a batch. This one zooms out. An LLM inference engine is a small distributed system whose job is to turn a stream of text requests into a stream of tokens while keeping an expensive accelerator as busy as possible. To reason about it you need the shape of the whole pipeline: how a request is tokenized, run through a compute-bound prefill and a memory-bound decode loop, then detokenized back to text — and which engine components (scheduler, KV manager, model runner, sampler) own each hand-off. This piece is the map. Where a stage has real math behind it, we point to the sibling article that derives it.
What an inference engine actually does
A training system optimizes weights; an inference engine spends them. Given fixed weights, its only job is to answer requests — but the economics are brutal, because the accelerator is the most expensive resource in the building and every idle millisecond is wasted money. So the engine is really a scheduling problem wrapped around a matrix multiply: keep the GPU saturated, respect per-request latency targets, and never run out of memory.
Two facts dominate the design. First, generation is autoregressive — token t+1 depends on token t, so a 1,000-token reply cannot be computed in one shot. Second, the two phases of a request have opposite hardware profiles: reading the prompt is compute-bound, while emitting each new token is memory-bandwidth-bound. Almost every architectural trick in the stack exists to reconcile those two facts.
The request lifecycle at a glance
Every request walks the same four-stage path, regardless of model size or serving framework:
text → [ tokenize ] → token ids
→ [ PREFILL ] → first token + KV cache
→ [ DECODE loop ] → one token per step, N times
→ [ detokenize ] → text out (streamed)Prefill runs once and processes the entire prompt in a single forward pass. The decode loop then runs once per output token, each step reading everything generated so far. The KV cache is the state that connects the two: prefill writes it, decode reads and extends it. Detokenization streams partial text back as tokens land, so the user sees output long before the reply finishes. The rest of the article fills in who runs each box and how they are kept busy.
Tokenize and detokenize: the text boundary
The model never sees characters — it sees integer token ids. The tokenizer (BPE, WordPiece, or a SentencePiece unigram model) maps the prompt to ids before anything numeric happens, and maps generated ids back to UTF-8 text on the way out. This looks trivial but has sharp edges: a single token can be a partial multi-byte character, so a naive detokenizer that emits token-by-token will print broken glyphs.
Streaming engines therefore buffer incomplete byte sequences and only flush when a full character is available. Tokenization also decides your effective context cost: prompt length in tokens, not words, drives both the prefill compute and the KV cache size.
Prefill: the one-shot prompt pass
Prefill is a single forward pass over the whole prompt. Because all prompt positions are known up front, they run in parallel through the layers as one big matrix multiply — which is why prefill is compute-bound and scales with prompt length times model FLOPs. Its two outputs are the first generated token and the initial KV cache: the per-layer key and value tensors for every prompt position, which decode will reuse instead of recomputing.
Prefill dominates time-to-first-token, so long prompts feel slow to start even on a fast model. Engines fight this with chunked prefill, which splits a long prompt into slices that interleave with ongoing decode work so one huge prompt cannot monopolize the GPU. The compute math — the O(n²) attention cost and how flash attention keeps memory linear — is in the prefill math article and chunked prefill.
The decode loop: autoregressive stepping
Decode is where most wall-clock time goes on long replies. Each step feeds the most-recent token through the model, attends over the entire cached K/V state, produces logits, samples one new token, and appends that token’s K/V to the cache. Then it repeats — once per output token — until an end-of-sequence token or the length limit stops it.
The defining property is that each step does a tiny amount of compute (one token) while reading a large amount of state (all cached keys and values plus the weights). That makes decode memory-bandwidth-bound: the GPU moves bytes rather than multiplying, so a single decode stream leaves most of the accelerator idle. That idleness is precisely why batching many concurrent decodes together is the central throughput lever, as the prefill-vs-decode deep dive quantifies.
The KV cache: state that ties the phases together
The KV cache is the single most important data structure in the engine. Without it, every decode step would re-attend over the full sequence from scratch, turning generation into an O(n²) disaster. With it, each step only computes the new token’s query against stored keys and values, so the per-step cost stays roughly flat as the sequence grows.
The price is memory. Cache size grows with batch size, sequence length, layers, heads, head-dimension, two (K and V), and the dtype width, and on long-context workloads it can dwarf the weights themselves. That pressure is why the cache is managed rather than simply allocated, and why quantizing it to 8- or 4-bit is common. Sizing is in the KV cache article; compression is in KV quantization.
Engine component: the scheduler
The scheduler is the brain. On modern engines it operates at the iteration level, not the request level: before every forward pass it decides which requests are in the batch, admits newly arrived ones the instant a slot frees, and preempts or pauses requests when memory runs short. This is what lets a fast request finish and leave while slower neighbors keep running, instead of the whole batch waiting for the longest member.
The scheduler also mixes phases — interleaving compute-bound prefill chunks with memory-bound decode steps so neither resource sits idle — and enforces priority and latency targets across tenants. The mechanics are in iteration scheduling, priority scheduling, and SLO scheduling math.
Engine component: the KV manager
If the scheduler decides who runs, the KV manager decides where their state lives. Naively giving each request a contiguous slab sized to its maximum length wastes memory, because most requests never reach that length — internal fragmentation routinely burned half the cache in early systems.
Paged attention fixes this by borrowing the operating-system idea of virtual memory: the cache is carved into small fixed-size blocks, and each request holds a block table mapping its logical positions to physical blocks allocated on demand. Fragmentation collapses to well under 5%, and two requests that share a prefix — the same system prompt, say — can point at the same physical blocks for free. The manager owns allocation, eviction, and that block table; the full mechanism is in paged attention.
Engine component: the model runner
The model runner is the piece that actually executes the network: it takes the batch the scheduler assembled, gathers the right KV blocks from the manager, and launches the forward pass. On a model too large for one device it also owns the parallelism — splitting each layer across GPUs with tensor parallelism, the layer stack across GPUs with pipeline parallelism, and driving the collective operations (all-reduce, all-gather) that stitch the shards together.
A 70-billion-parameter model at 16-bit is ~140 GB of weights, which does not fit on one 80 GB device, so at that scale sharding is not optional — it is why a single request touches several GPUs and an interconnect. The runner is also where fused kernels, flash attention, and quantized matmuls plug in, because it is the only component that speaks directly to the hardware.
Engine component: the sampler
The forward pass ends in a vector of logits — one score per vocabulary token. The sampler turns that vector into the next token. Greedy decoding picks the argmax; temperature rescales the distribution; top-k and top-p (nucleus) truncate it to a plausible set before sampling; repetition and presence penalties nudge it away from loops. These knobs separate a robotic reply from a fluent one.
The sampler runs every decode step, so it sits on the hot path, but its cost is tiny next to the matrix multiplies before it. It is also where constrained decoding lives: grammar or JSON-schema masks that zero out illegal tokens before sampling to guarantee well-formed output. The probability math for each strategy is in the sampling math article.
Continuous batching: keeping the GPU full
The reason all of these components can coexist is continuous batching. Classic static batching gathers a fixed group of requests, runs them to completion together, and only then accepts the next group — so the whole batch idles at the pace of its slowest member. Continuous batching instead treats the batch as fluid: at every iteration finished requests drop out and waiting requests slot in.
Because decode is memory-bound and a lone stream barely tickles the GPU, packing many decodes into one step is nearly free on compute and multiplies throughput several-fold — the biggest serving win of recent years. The throughput arithmetic is derived in continuous batching math.
Disaggregated prefill and decode
Because prefill is compute-bound and decode is memory-bound, running both on the same GPU forces a compromise: a big prefill stalls the decodes sharing its device. Disaggregated serving splits them onto separate pools — prefill workers optimized for throughput, decode workers optimized for latency — and hands the freshly built KV cache across the network from one pool to the other.
The win is that each pool can be sized, batched, and even hardware-matched independently, which stabilizes tail latency under mixed load. The cost is a cache transfer per request and a more complex control plane, so it pays off mainly at scale. It is the clearest example of the architecture’s organizing principle: let each phase run on the resource it is actually bound by.
Reading the stack as one system
Put the pieces together and a request’s journey is legible. Text is tokenized; the scheduler admits it into an iteration; the KV manager reserves blocks; the model runner prefills the prompt and writes the cache; the sampler emits the first token; then the decode loop turns — scheduler batching it with dozens of neighbors, manager growing its block table, runner executing each step, sampler choosing each token — while the detokenizer streams text back.
The recurring theme is that no single stage is the bottleneck; the hand-offs are. Time-to-first-token is a prefill and scheduling story; tokens-per-second is a decode, batching, and bandwidth story; cost-per-token is whether the KV manager and scheduler together keep the accelerator full. To diagnose a slow endpoint, ask which hand-off is stalling.