Prefill/Decode Disaggregation Architecture in Depth

Two phases, two rooflines

Put both phases on a roofline and they land on opposite sides of the ridge point. Prefill multiplies a weight matrix against S prompt tokens at once, and every weight byte fetched from HBM is reused across all S rows. Arithmetic intensity therefore scales with prompt length, and a few thousand tokens is already deep in the compute-bound regime. The kernels look like training kernels: large GEMMs, high tensor-core utilization, HBM bandwidth largely idle.

Decode is the mirror image. Each step processes one new token per sequence, so a batch of B sequences turns every weight matrix into a skinny GEMM with B rows. Intensity is proportional to B, not to context length, and at the batch sizes most services run it sits well below the ridge point: step time is set by how fast you stream weights and KV out of HBM. Attention makes it worse, since decode reads every cached key and value for every active sequence.

Interference — one long prompt stalls everyone

The practical consequence of colocating the two phases is head-of-line blocking. A GPU running a decode loop executes one short step after another, each producing a token for every sequence in the batch. Drop a 32k-token prefill into that loop and the GPU disappears for the duration of that forward pass. Every sequence in the batch, including ones that have been streaming smoothly for a minute, sees its next token arrive late.

That is the shape of the problem: inter-token latency for all users is hostage to the longest prompt any user submits. Median ITL can look excellent while p99 is a mess of periodic stalls, and tuning the batch size does not fix it, because the stall is one indivisible kernel sequence rather than a queueing artifact. Prefill also wants the opposite scheduling policy from decode: the whole machine briefly, versus a steady predictable slice forever. One scheduler cannot honour both.

What disaggregation actually changes

Disaggregation resolves the conflict by refusing to schedule the two phases against each other at all. One pool of GPUs runs prefills and nothing else; a second runs decode steps and nothing else. A router sends a request to a prefill worker, that worker produces the KV cache for the prompt plus the first token, the cache is transferred to a decode worker, and the decode worker admits the request into its running batch.

The important word is isolation, not specialization. Both pools normally run the same GPU SKU; what differs is what each is allowed to do. The decode loop now has a hard guarantee that no prefill will land inside it, so its step time depends only on its own batch and context size. The prefill pool, freed from yielding, runs forward passes back to back at whatever batch shape maximizes tensor-core throughput.

The handoff is the KV cache, and it is big

The state a request carries between the pools is its KV cache. Size it before anything else, because it decides whether the architecture is viable at all. Per token the cache holds a key and a value vector for every layer:

bytes/token = 2 × layers × kv_heads × head_dim × bytes_per_element

Take an illustrative 70B-class model: 80 layers, grouped-query attention with 8 KV heads, head dimension 128, FP16. That is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, about 320 KiB per token, so an 8k-token prompt hands off roughly 2.5 GiB. Multi-head models without GQA are several times larger; FP8 caches roughly halve it. This is not a control message, it is a multi-gigabyte tensor that must cross from one GPU's HBM to another's inside the user's time-to-first-token budget.

The interconnect decides whether this works

Divide cache size by the usable bandwidth between the pools and you get the transfer time added directly to TTFT. Inside a node, NVLink gives hundreds of GB/s per direction and a couple of gigabytes moves in single-digit milliseconds — free, relative to a prefill that took tens of milliseconds. Across nodes over InfiniBand or RoCE at a few tens of GB/s effective, the same payload takes tens of milliseconds. Over PCIe it is worse still.

The deployment rules follow from the fabric. Use GPUDirect RDMA so the transfer goes NIC-to-HBM without staging in system RAM. Pin the buffers. Keep a prefill worker and its decode partner rail-aligned or inside one NVLink domain. And overlap: the cache is produced layer by layer, so a worker can stream layer L's keys and values while computing layer L+1, hiding most of the transfer behind the prefill. Without RDMA and overlap, disaggregation often loses on TTFT even when it wins on ITL.

Sizing the two pools

Pool sizing is a ratio problem, and the ratio is set by how much GPU time each phase consumes per request. Prefill work is roughly 2 × P × S FLOPs for P parameters and S prompt tokens, so its cost tracks prompt length. Decode cost is bandwidth, not FLOPs: each step streams the weights once, so a request holds a slot in the decode batch for as many steps as it generates tokens.

The arithmetic follows. Measure the GPU-seconds a representative request spends prefilling and the GPU-seconds it spends decoding, then provision the pools in that proportion. A summarization workload with 16k prompts and 200-token answers wants a fat prefill pool; a chat workload with short prompts and long streamed replies inverts it. The ratio is a property of your traffic mix, not of the model, so derive it from your own token histograms and re-derive it when traffic shifts.

Independent scaling and separate SLOs

Once the pools are separate, the two latency metrics decouple. Time-to-first-token is a property of the prefill pool plus the transfer; inter-token latency is a property of the decode pool alone. You can autoscale each on the signal that reflects its own pressure — prefill queue depth against a TTFT objective, decode batch occupancy and KV-cache utilization against an ITL objective — instead of tuning one knob that trades them against each other.

The pools can also be configured differently. Prefill is latency-sensitive per request and can justify heavier tensor parallelism to shorten a single forward pass. Decode benefits more from aggregate memory capacity, since KV cache limits how many sequences a worker can hold, and from parallelism choices that keep per-step collectives cheap. Two pools mean two independent answers to the parallelism question instead of one compromise.

The costs you sign up for

Nothing here is free. Model weights must be resident in both pools, so every worker in both halves pays the same parameter-memory tax and the smallest sensible deployment is several GPUs rather than one. The transfer adds latency to TTFT and load to a network that is often already the scarce resource. The router becomes stateful and load-bearing: it must know which decode workers have KV capacity, and a bad placement means a transfer across the slowest path in the cluster.

Failure modes multiply. If the decode pool saturates, completed prefills pile up holding KV memory they cannot hand off, so back-pressure must propagate to the router or the prefill pool fills with orphaned state. Losing a decode worker kills every generation on it and the prompt must be prefilled again. And ratio drift is the quiet one: one pool idles while the other queues.

When chunked prefill on one pool wins

There is a cheaper fix for the interference problem, and it is often the right one. Chunked prefill splits a long prompt into fixed token-budget pieces and lets the scheduler put one chunk into the same batch as ongoing decode steps. The long forward pass is no longer indivisible, so the ITL stall is bounded by chunk size instead of prompt length — and the mixed batch gives bandwidth-starved decode work some compute-heavy company to share the machine with.

Chunked prefill wins whenever the second pool is the expensive part: small deployments where duplicating the weights doubles the GPU bill, clusters whose inter-GPU path is PCIe rather than NVLink or RDMA, short-prompt workloads where prefill never blocks long enough to matter, and any setup where you would rather not run a stateful router. One pool, one scheduler, one set of weights.

Choosing between them

Reach for disaggregation when four things are true together: prompts are long enough that prefill dominates a step, ITL has a tight tail objective chunking alone cannot hold, the fleet is large enough that a few percent of utilization is real money, and the fabric between the pools is NVLink or RDMA-capable. Systems like DistServe, Splitwise, Mooncake and NVIDIA Dynamo exist because at that scale the isolation is worth the transfer and the operational weight.

Otherwise reach for chunked prefill on a single pool, which is the right answer most of the time for most teams. Disaggregation is not a latency trick, it is a goodput trick: more requests served within both SLOs per GPU, because each phase is scheduled and scaled on its own terms. If you cannot measure that gain against your own traffic, and cannot pay the KV transfer at fabric speed, the second pool is complexity you have not earned yet.