Speculative decoding architecture — draft, verify, tree, Medusa, self-spec

Why architecture matters here

Serving cost dominates LLM economics. If you can process 3 tokens per forward pass instead of 1, your GPU serves 3× the traffic at the same quality. The architecture matters because getting speculation right requires careful tuning: draft model choice, k (lookahead depth), accept criteria, and integration with continuous batching.

Wrong choices give small or negative gains. A draft model that disagrees with the target loses more time to rejection than it saves. Excessive k thrashes the KV cache. Tree speculation without the right verifier changes semantics.

Realized speedup is a ratio you can compute, not a constant you can quote: tokens committed per verification step, divided by the cost of that step in target-forward equivalents.

The architecture: every piece explained

The top strip is the base loop. Prompt comes in. Draft model is a small, fast model — often the same family, smaller size (e.g. 1B alongside a 70B target). It generates Speculative tokens — the next k proposals with their probabilities. The Target model then processes the prompt + speculative tokens in a single forward pass, computing target-model logits at every position simultaneously.

The middle row is the correctness machinery. Accept-reject compares target probability to draft probability for each proposed token; accept if target agrees with a probabilistic rule that preserves the target's distribution. The longest accepted prefix is committed. Tree speculation proposes multiple candidate paths (a tree) and verifies all in one pass, increasing accepted length per pass. Medusa heads attach small prediction heads to the target model that predict the next 2-4 tokens in one pass. Self-spec uses n-gram lookup or lookahead within the target model to generate speculative candidates without a separate draft model.

The lower rows are integration and observability. Accept rate is the quality signal — high accept rate means draft matches target; low means the speculation is not paying off. Serving integration is how speculative decoding fits into vLLM, TGI, or TRT-LLM's continuous batching. Metrics track mean accepted length, effective tokens per second, and cost per token.

End-to-end flow

End-to-end: a user asks the 70B target model a question. The 1B draft runs on the same prefix and proposes five tokens, costing five sequential draft forwards. The target processes prefix plus the five candidates in one pass and obtains its own distribution at each of the six positions. The acceptance rule walks the five in order; say it accepts three and rejects the fourth. Position four is filled from the residual distribution, the fifth proposal is discarded along with its KV entries in both models, and the step has committed four tokens. Whether that beats four ordinary decode steps depends on the draft's cost ratio and on whether the target step was bandwidth bound to begin with. The production metric is mean accepted length per verification step, sliced by draft/target pair and request type.

The exact acceptance rule - and the errors that break it

Draft-then-verify would be worthless if it changed what the model says, and it does not. Fix the notation first. At draft position i the draft sampled token x_i from its own distribution q_i. The single target forward returns p_i, the target's distribution at that same position conditioned on the prefix plus the first i-1 drafted tokens - and it returns all of them at once, because the causal mask already makes each position see only what precedes it. That is the whole trick: one pass yields the distributions a sequential run would have produced, on the assumption that the drafts were accepted.

The rule then walks left to right and stops at the first rejection:

for i in 1..gamma:
    r ~ Uniform(0, 1)
    if r <= min(1, p_i(x_i) / q_i(x_i)):
        emit x_i                                  # accepted
    else:
        emit y ~ normalize(max(0, p_i - q_i))     # residual distribution
        discard x_i .. x_gamma                    # the tail is now invalid
        break
else:                                             # all gamma survived
    emit y ~ p_(gamma+1)                          # bonus token, already computed

The proof is two lines. The accept route emits token x with probability q(x) * min(1, p(x)/q(x)), which is min(p(x), q(x)). The reject route fires with probability 1 - sum_x min(p, q) and then draws from a residual whose normalizing constant is exactly that same quantity, so the normalizers cancel and the route contributes max(0, p(x) - q(x)). Add them: min(p, q) + max(0, p - q) = p(x). The emitted token is a sample from the target at every position, for any draft, however bad it is. A poor draft costs time, never quality. Full derivation: Speculative Decoding - The Math.

Four implementation errors break that guarantee. Comparing top tokens - "did the draft's argmax match the target's argmax" - is equivalent to the rule above only at temperature zero; above it, exact-match verification is a different and biased sampler. Resampling from p on rejection, instead of from the residual, double-counts the mass the accept route already delivered. Keeping the tail past a rejection emits tokens conditioned on a prefix that never happened; they and their KV entries have to go. Mismatched samplers fail silently: p and q must be the distributions actually sampled from, after temperature, top-k, top-p and repetition penalties, or you will accept tokens the target's own sampler would never emit. See sampling and decoding for what those processors do to the distribution.

Acceptance rate, draft cost, and the speedup that follows

One number governs the technique: alpha, the average probability that a drafted token survives the rule. It is the complement of the total-variation distance between draft and target at that position, so it measures agreement, not draft quality in the abstract - a weak draft that is wrong in the same way the target is wrong scores well.

Under the usual geometric idealization, one verification step emits (1 - alpha^(gamma+1)) / (1 - alpha) tokens. Read that carefully, because the off-by-one is itself a common error: it counts the accepted drafts plus one more token, either the residual resample at the rejection point or the bonus token when every draft survives. It is therefore never below 1. A speculative step always makes at least as much progress as a plain decode step, which is why speculation can cost you time but never tokens.

Time is the other half. If a draft forward costs a fraction c of a target forward, the step costs about gamma * c + 1 target-equivalents and realized speedup is the ratio. The ceiling is 1 / (1 - alpha) however large gamma grows, so closing the draft/target gap beats lengthening the speculation. And that + 1 encodes an assumption - that verifying gamma+1 positions costs the same as decoding one token - which holds only while the target step is memory-bandwidth bound. Algebra in acceptance rate and optimal draft length; the bandwidth argument in bandwidth-bound operations.

Choosing the draft, and tuning the speculation length

What raises alpha: a shared tokenizer, effectively mandatory since mismatched vocabularies mean drafted tokens do not align with target positions at all; a draft distilled on the target's own outputs rather than on generic corpora; and content whose next token is structurally forced - closing brackets, JSON keys, repeated identifiers, boilerplate. What lowers it: proper nouns, digits, the first token after a clause boundary, unseen domains. Acceptance is not uniform along a response. It is high through the predictable stretches and collapses at exactly the decision points that carry the information.

What lowers c - fewer layers, smaller hidden size, a quantized draft - usually lowers alpha too, and the two enter the speedup differently: c linearly, alpha through a geometric sum with a hard ceiling. The gamma optimum sits near where alpha^gamma decays to c, so low acceptance means short speculation. That optimum is flat, which is why a per-request gamma, a running acceptance estimate, or simply letting the draft stop when its own top-1 probability drops below a threshold all beat a carefully tuned global constant.

Draft-free variants change the arithmetic rather than the rule. N-gram or prompt-lookup drafting matches the last few tokens against the prompt and output so far and proposes what followed last time; its c is essentially zero, so rejects cost only wasted verify slots and it is safe to leave enabled, and its alpha is high exactly where output copies input. Self-speculation reuses the target's own early layers as the drafter - no extra weights, one more KV bookkeeping path. Medusa-style heads predict positions t+2 and beyond from the target's final hidden state; being independent they give no properly factorized joint, so those systems verify a tree of head combinations under a mask that lets each node attend only to its ancestors, and relax exact rejection sampling into typical acceptance - trading the distributional guarantee for accept rate. Mechanics in Medusa head math and tree attention and KV rollback.