Backpressure Math

Queue depth is an integral, not a level

The most useful equation in this topic is a first-order differential one. If work arrives at rate λ(t) and drains at rate μ(t):

dQ/dt = λ(t) − μ(t)
Q(t)  = Q(0) + ∫_0^t [ λ(s) − μ(s) ] ds

Two consequences follow. A queue responds not to the level of load but to the mismatch: 1% overload and 50% overload both grow without bound, just at different slopes. And growth is linear, not explosive — what people call ‘the queue blew up’ is a straight line of slope λ − μ crossing a threshold that matters. Linear is predictable, which is the good news. The unforgiving corollary is that an integral has memory: clearing the backlog built by 60 s of 10% overload needs a later period of underload with equal area, so the damage outlives the spike.

Little's law, applied stage by stage

Little’s law states that for any stable system L = λ · W — average occupancy equals arrival rate times average residence time. Its power here is that it holds for every stage independently when the same flow passes through, so end-to-end latency decomposes:

W_total = Σ_i W_i = Σ_i ( L_i / λ )

Each stage’s contribution to latency can therefore be read straight off a queue-depth gauge. Run it backwards and it becomes a sizing rule: if a stage drains at μ and you will tolerate at most W_max seconds of waiting there, its bound must be L_max = μ · W_max. That is the honest way to choose a queue size. A bound of 1024 chosen because it is a round number is a latency budget chosen by accident.

A chain of bounded buffers in mismatched units

A request crossing an LLM server passes through at least four buffers — the client’s in-flight cap, the gateway queue, the engine’s waiting queue, the running batch — then out through a socket buffer. Pressure gets lost in translation between them because they do not share units. A bound expressed in requests says nothing about the resource that actually runs out: sixteen queued requests with 200-token prompts and sixteen with 100k-token prompts are the same number and a 500× difference in work. Any signal that survives the trip upstream must be re-expressed in the units of the stage receiving it — KV blocks become an admission decision, refusal becomes a full queue, a full queue becomes a 429, and a 429 becomes a smaller client concurrency window.

The prefill/decode token-rate mismatch

Inside the engine one request is really two workloads. Prefill consumes the prompt in a compute-bound pass costing roughly linearly in prompt length P; decode emits D tokens one step at a time, bound by memory bandwidth. Arrivals convert into two separate demands:

prefill demand = λ · P   [prompt tok/s]
decode  demand = λ · D   [output tok/s]

Either can saturate alone. Long prompts with short answers saturate prefill while decode idles; chat traffic does the reverse. And because both are drawn from one shared per-step token budget, saturating one starves the other: a prefill flood does not merely delay new requests, it steals decode steps from sequences already streaming, so inter-token latency rises for existing users. That is backpressure leaking sideways into a different SLO.

A worked example: how fast does the backlog build?

Take a replica measured at 6,000 prompt tok/s of prefill and 2,400 output tok/s of aggregate decode. Offer it λ = 8 req/s, mean P = 800, D = 200:

prefill: 8 × 800 = 6,400 tok/s  vs 6,000  → OVER by 400
decode : 8 × 200 = 1,600 tok/s  vs 2,400  → 33% headroom

μ in requests = 6,000 / 800 = 7.5 req/s
dQ/dt = 8.0 − 7.5 = 0.5 req/s

Decode dashboards look healthy while the system fails. After 60 s the backlog is 30 and the wait is W = L/μ = 30/7.5 = 4 s; after five minutes, 150 queued and 20 s of wait. Now invert for the bound: if time-to-first-token may spend at most 5 s queued, L_max = 7.5 × 5 ≈ 37. Queue 37, shed the 38th — a number derived rather than guessed.

KV memory: the buffer that tightens under you

The running batch has no configurable length; its bound is physical. Each resident sequence holds 2 · L · H_kv · d_head values per token of context, so capacity is KV_bytes / (per-token bytes · context) — and that denominator grows every decode step. A batch that fit at admission can stop fitting 2,000 tokens later: the bottleneck tightens while work is in flight. When KV runs out the engine preempts, swapping a sequence out or evicting and recomputing its prefix. Preemption is negative service rate, pushing work from running back to waiting and burning compute to do it, so μ drops exactly when λ is highest.

Continuous batching then disguises the resulting pressure. Instead of making you wait your turn, the engine admits you and slows everyone together, per-sequence throughput falling roughly as μ_seq ≈ μ_total / B. So overload shows up as degraded inter-token latency, not queue depth — and since L = λW with rising W means rising occupancy, which means more KV pressure, the loop feeds itself. That is why engines fall off a cliff rather than degrade gently, and why a per-step token budget beats a batch-size cap.

Slow consumers: backpressure arriving from the client

Streaming inverts the usual direction. If a client reads a server-sent-event stream at 20 tok/s while the model produces 60 tok/s, the send buffer fills, the write blocks, and the sequence cannot retire — it sits in the batch holding KV blocks while doing no useful work. Residence time inflates by the ratio of the rates:

W’ = W · (60/20) = 3W   →   L’ = λW’ = 3L

Three times the KV footprint at an unchanged arrival rate, caused entirely by a slow reader. Mitigations are all forms of refusing to hold the bag: cap the per-stream output buffer, check the client-disconnect signal every step rather than at the end and cancel generation when it fires, and impose a hard max_tokens so nothing can occupy KV indefinitely.

Keeping the signal alive on the way upstream

A pressure signal is worth only as much as the weakest link in its chain, and every hop needs an explicit mechanism: bounded channels between internal stages so a full consumer blocks its producer, HTTP/2 or gRPC flow-control windows so a stalled reader stalls the writer, 429 with Retry-After at the gateway, and a client that honours it with capped concurrency and jittered backoff.

The rule is that every stage must be able to say no to the one before it, and a single unbounded buffer anywhere breaks all of them. This is bufferbloat in new costume: an unbounded queue does not prevent loss, it converts loss into latency — and latency past the caller’s timeout is loss anyway, loss you paid full compute for. A 40-deep queue that rejects beats a 10,000-deep one that accepts.

Shed on delay, not on depth

Queue length is the wrong shedding trigger because its meaning depends on a service rate that moves: 40 queued is comfortable at μ = 20/s and catastrophic at μ = 1/s. Queue sojourn time is the invariant, and it is directly comparable to the SLO. The CoDel discipline generalizes cleanly: stamp each request on enqueue and start shedding when the minimum sojourn observed over a sliding interval stays above target for longer than that interval.

Two refinements pay for themselves. Drop work that is already dead — if the enqueue stamp shows the caller’s timeout expired, running it yields zero goodput at full cost, the textbook shape of congestion collapse. And consider LIFO under overload: FIFO guarantees that once the backlog exceeds the timeout every served request is stale, while LIFO turns a total outage into a partial one.

CPU SLM serving: small numbers, sharp edges

On a CPU-hosted small model every quantity shrinks, and the shrinking is what makes it dangerous. Aggregate decode throughput might be 30–80 tok/s rather than thousands, so μ in requests per second is often below one. A single 8k-token prompt can hold the box for seconds; with one shared thread pool that is strict head-of-line blocking, μ → 0 for the duration and dQ/dt ≈ λ.

The bounds come out startlingly small. At μ = 0.8 req/s with a 6 s budget, L_max = 0.8 × 6 ≈ 5. Five. Operators leave a default of several hundred in place and wonder why p99 is measured in minutes. And thread oversubscription makes μ fall as concurrency rises, so the queue drains slowest when it is deepest.

Pitfalls that quietly disable backpressure

Most broken pipelines have a mechanism that looks present and is not.

Timeouts that do not cancel. A caller giving up while the server keeps generating means work continues, KV stays held, and the retry adds fresh load. A timeout without propagated cancellation is an amplifier, not a limit.

Retrying 429 like it were 500. An immediate retry multiplies λ at precisely the wrong moment, converting the pressure signal into more pressure. Honour Retry-After, add jitter, and cap retries as a percentage of traffic.

Health checks that pass under overload. A cheap /healthz answers instantly while the inference queue is 200 deep, so the balancer keeps feeding a replica that cannot cope. Health must reflect sojourn time.

Averaging the mismatch away. A one-minute mean hides a 10 s burst whose backlog costs a minute to repay — because Q is an integral, transient overload is durable latency.