Speculative decoding architecture — draft, verify, tree, Medusa, self-spec

Why architecture matters here

Serving cost dominates LLM economics. If you can process 3 tokens per forward pass instead of 1, your GPU serves 3× the traffic at the same quality. The architecture matters because getting speculation right requires careful tuning: draft model choice, k (lookahead depth), accept criteria, and integration with continuous batching.

Wrong choices give small or negative gains. A draft model that disagrees with the target loses more time to rejection than it saves. Excessive k thrashes the KV cache. Tree speculation without the right verifier changes semantics.

Realized speedup is a ratio you can compute, not a constant you can quote: tokens committed per verification step, divided by the cost of that step in target-forward equivalents.

The architecture: every piece explained

The top strip is the base loop. Prompt comes in. Draft model is a small, fast model — often the same family, smaller size (e.g. 1B alongside a 70B target). It generates Speculative tokens — the next k proposals with their probabilities. The Target model then processes the prompt + speculative tokens in a single forward pass, computing target-model logits at every position simultaneously.

The middle row is the correctness machinery. Accept-reject compares target probability to draft probability for each proposed token; accept if target agrees with a probabilistic rule that preserves the target's distribution. The longest accepted prefix is committed. Tree speculation proposes multiple candidate paths (a tree) and verifies all in one pass, increasing accepted length per pass. Medusa heads attach small prediction heads to the target model that predict the next 2-4 tokens in one pass. Self-spec uses n-gram lookup or lookahead within the target model to generate speculative candidates without a separate draft model.

The lower rows are integration and observability. Accept rate is the quality signal — high accept rate means draft matches target; low means the speculation is not paying off. Serving integration is how speculative decoding fits into vLLM, TGI, or TRT-LLM's continuous batching. Metrics track mean accepted length, effective tokens per second, and cost per token.

End-to-end flow

End-to-end: a user asks the 70B target model a question. The 1B draft runs on the same prefix and proposes five tokens, costing five sequential draft forwards. The target processes prefix plus the five candidates in one pass and obtains its own distribution at each of the six positions. The acceptance rule walks the five in order; say it accepts three and rejects the fourth. Position four is filled from the residual distribution, the fifth proposal is discarded along with its KV entries in both models, and the step has committed four tokens. Whether that beats four ordinary decode steps depends on the draft's cost ratio and on whether the target step was bandwidth bound to begin with. The production metric is mean accepted length per verification step, sliced by draft/target pair and request type.

The exact acceptance rule - and the errors that break it

Draft-then-verify would be worthless if it changed what the model says, and it does not. Fix the notation first. At draft position i the draft sampled token x_i from its own distribution q_i. The single target forward returns p_i, the target's distribution at that same position conditioned on the prefix plus the first i-1 drafted tokens - and it returns all of them at once, because the causal mask already makes each position see only what precedes it. That is the whole trick: one pass yields the distributions a sequential run would have produced, on the assumption that the drafts were accepted.

The rule then walks left to right and stops at the first rejection:

for i in 1..gamma:
    r ~ Uniform(0, 1)
    if r <= min(1, p_i(x_i) / q_i(x_i)):
        emit x_i                                  # accepted
    else:
        emit y ~ normalize(max(0, p_i - q_i))     # residual distribution
        discard x_i .. x_gamma                    # the tail is now invalid
        break
else:                                             # all gamma survived
    emit y ~ p_(gamma+1)                          # bonus token, already computed

The proof is two lines. The accept route emits token x with probability q(x) * min(1, p(x)/q(x)), which is min(p(x), q(x)). The reject route fires with probability 1 - sum_x min(p, q) and then draws from a residual whose normalizing constant is exactly that same quantity, so the normalizers cancel and the route contributes max(0, p(x) - q(x)). Add them: min(p, q) + max(0, p - q) = p(x). The emitted token is a sample from the target at every position, for any draft, however bad it is. A poor draft costs time, never quality. Full derivation: Speculative Decoding - The Math.

Four implementation errors break that guarantee. Comparing top tokens - "did the draft's argmax match the target's argmax" - is equivalent to the rule above only at temperature zero; above it, exact-match verification is a different and biased sampler. Resampling from p on rejection, instead of from the residual, double-counts the mass the accept route already delivered. Keeping the tail past a rejection emits tokens conditioned on a prefix that never happened; they and their KV entries have to go. Mismatched samplers fail silently: p and q must be the distributions actually sampled from, after temperature, top-k, top-p and repetition penalties, or you will accept tokens the target's own sampler would never emit. See sampling and decoding for what those processors do to the distribution.

Acceptance rate, draft cost, and the speedup that follows

One number governs the technique: alpha, the average probability that a drafted token survives the rule. It is the complement of the total-variation distance between draft and target at that position, so it measures agreement, not draft quality in the abstract - a weak draft that is wrong in the same way the target is wrong scores well.

Under the usual geometric idealization, one verification step emits (1 - alpha^(gamma+1)) / (1 - alpha) tokens. Read that carefully, because the off-by-one is itself a common error: it counts the accepted drafts plus one more token, either the residual resample at the rejection point or the bonus token when every draft survives. It is therefore never below 1. A speculative step always makes at least as much progress as a plain decode step, which is why speculation can cost you time but never tokens.

Time is the other half. If a draft forward costs a fraction c of a target forward, the step costs about gamma * c + 1 target-equivalents and realized speedup is the ratio. The ceiling is 1 / (1 - alpha) however large gamma grows, so closing the draft/target gap beats lengthening the speculation. And that + 1 encodes an assumption - that verifying gamma+1 positions costs the same as decoding one token - which holds only while the target step is memory-bandwidth bound. Algebra in acceptance rate and optimal draft length; the bandwidth argument in bandwidth-bound operations.

Choosing the draft, and tuning the speculation length

What raises alpha: a shared tokenizer, effectively mandatory since mismatched vocabularies mean drafted tokens do not align with target positions at all; a draft distilled on the target's own outputs rather than on generic corpora; and content whose next token is structurally forced - closing brackets, JSON keys, repeated identifiers, boilerplate. What lowers it: proper nouns, digits, the first token after a clause boundary, unseen domains. Acceptance is not uniform along a response. It is high through the predictable stretches and collapses at exactly the decision points that carry the information.

What lowers c - fewer layers, smaller hidden size, a quantized draft - usually lowers alpha too, and the two enter the speedup differently: c linearly, alpha through a geometric sum with a hard ceiling. The gamma optimum sits near where alpha^gamma decays to c, so low acceptance means short speculation. That optimum is flat, which is why a per-request gamma, a running acceptance estimate, or simply letting the draft stop when its own top-1 probability drops below a threshold all beat a carefully tuned global constant.

Draft-free variants change the arithmetic rather than the rule. N-gram or prompt-lookup drafting matches the last few tokens against the prompt and output so far and proposes what followed last time; its c is essentially zero, so rejects cost only wasted verify slots and it is safe to leave enabled, and its alpha is high exactly where output copies input. Self-speculation reuses the target's own early layers as the drafter - no extra weights, one more KV bookkeeping path. Medusa-style heads predict positions t+2 and beyond from the target's final hidden state; being independent they give no properly factorized joint, so those systems verify a tree of head combinations under a mask that lets each node attend only to its ancestors, and relax exact rejection sampling into typical acceptance - trading the distributional guarantee for accept rate. Mechanics in Medusa head math and tree attention and KV rollback.

Where speculation sits in the serving stack

This is where speculative decoding stops being an algorithm and becomes an operational problem. A speculating sequence needs gamma+1 KV slots reserved for the coming step, not one - under paged KV, up to gamma+1 extra block-table entries per sequence per step. Under-reserving produces an allocation failure mid-step, which the engine resolves by preempting a request, turning a throughput optimization into a latency incident. Admission control has to price a speculating sequence at its worst case. See PagedAttention for the block mechanics.

Rollback is the other bookkeeping. The target wrote KV entries for every verified position including the rejected ones, so the cache manager must treat the logical accepted length as truth and let later writes overwrite those slots. The draft's cache needs the same rewind, and a subtler one: back to the accepted length, not to where the draft stopped drafting. Getting that wrong leaves the draft conditioned on tokens the target discarded, which surfaces as a slowly decaying acceptance rate rather than as a crash. Draft weights and draft KV also come out of the same HBM budget as the target's cache, so enabling speculation shrinks the maximum concurrent batch before a single token is produced.

Two effects surprise operators. Tokens now arrive in bursts of one to gamma+1, so mean inter-token latency falls while its variance rises - a p99 inter-token SLA can regress at the same moment end-to-end completion time improves. And the guarantee is distributional, not sample-level: speculative output is a valid sample from the target, but not the sample a seeded non-speculative run produces, so golden-output regression suites diverge the day it is switched on.

Batch size decides whether any of this pays

At batch one, a decode step reads every weight in the model to produce a single token; arithmetic intensity is dismal and the tensor cores idle. Verifying gamma+1 positions turns that matrix-vector product into a skinny matrix-matrix product riding the same weight read, so the extra positions are nearly free. Speculation spends idle FLOPs to buy back wall-clock time.

As the batch grows, the weight read is already amortized across sequences and the step drifts toward compute bound. The extra positions become real work - B sequences each verifying gamma+1 tokens present B * (gamma+1) rows to every kernel - and each rejected token is FLOPs some other request could have used. There is a crossover load above which speculation lowers aggregate throughput while still improving single-request latency; the two goals genuinely conflict there.

Mixture-of-experts targets bend this further. One decode token activates only the experts its router picks, but gamma+1 positions in a single pass can route to a much larger union of experts, so the verify step reads more expert weight bytes than a plain decode step would. The "verifying is nearly free" premise erodes precisely on the architecture whose decode step is otherwise the most bandwidth-starved - measure it rather than assume it when the target is sparse (MoE serving). The conclusion is that speculation is a scheduling decision, not a deployment flag: on at low queue depth, gamma reduced as the batch fills, off above a threshold. That policy belongs in continuous batching, the only component that knows the current batch composition.

When speculative decoding does not help

Offline batch scoring and other throughput-optimized jobs already run compute bound; speculation only adds rejected-token FLOPs. Prompt-heavy, output-light work - classification, extraction, reranking - spends its time in prefill, which speculation does not touch at all (that ground belongs to chunked prefill and prefill/decode disaggregation). A deployment with no tokenizer-compatible small model has no cheap draft without paying for a distillation run, and a memory-tight server pays for one in lost batch capacity. High-temperature open-ended generation flattens the target distribution enough that acceptance falls below what gamma * c costs.

The favourable cases are the mirror image, and one is counter-intuitive: long-context decode is a good fit, because attention reads the whole KV cache once per step no matter how many query positions you push through it, so verifying gamma+1 positions amortizes that read as well as the weight read. Interactive single-stream serving, agent loops emitting structured calls, and any generation that quotes its own input are where acceptance is high and the batch is small at the same time.