Speculative decoding is usually explained as a proof: a small model guesses, a big model checks, and a rejection-sampling step guarantees the output distribution is exactly the big model’s. That is the companion article’s job. This one is about everything the proof leaves out — the engineering. Where does the draft actually come from? What happens to the KV cache when half a draft is thrown away? How do you verify five candidate continuations in one forward pass? And why does a technique that triples tokens per second on a laptop quietly become a slowdown on a saturated serving cluster? Speculative decoding is a memory-bandwidth trick, and every real design decision follows from that one fact.
The win is bandwidth, not cleverness
Autoregressive decoding at batch size 1 is memory-bandwidth bound. To emit one token you stream every weight in the model from memory into the compute units and do about two FLOPs per byte read; the arithmetic units sit almost entirely idle. Hence the fact that makes speculation possible: a forward pass over M tokens costs barely more wall-clock than a pass over 1, because the dominant term — sweeping the weights — is paid once either way.
The target therefore has free token-slots every step, and speculation’s whole job is to fill them with plausible guesses:
70B target, fp16 = 140 GB weights; HBM = 2.0 TB/s
target step ≥ 140/2000 s = 70 ms → ~14 tok/s ceiling
1B drafter, fp16 = 2 GB → ~1 ms/step
cycle = 5 draft steps (5 ms) + 1 verify (70 ms) = 75 ms
E[accepted] = 3.5 → 3.5 / 75 ms ≈ 47 tok/s (~3.3×)Every section below is a consequence of trying to raise that E[accepted] without raising the 70 ms.
Choosing a drafter, and the tokenizer trap
The classic setup is a separate small model from the same family: a 1B drafting for a 70B. Two hard constraints govern the choice. First, cost ratio — if the drafter costs a fraction c of a target step, a draft of length K adds K·c to every cycle, so c ≈ 0.01–0.05 is the workable band. A 7B drafting for a 13B is almost always a net loss.
Second, and more often fatal: the two models must share a tokenizer and vocabulary. Verification compares the target’s distribution p(x) against the drafter’s q(x) at the same position over the same token ids. Different tokenizers means different segmentations — the drafter’s three tokens may not correspond to any prefix of the target’s — so there is no position-aligned comparison to make. Even same-family jumps break this: Llama 2 has a 32,000-token vocabulary, Llama 3 has 128,256. Cross-tokenizer schemes exist (detokenize, re-tokenize, map over the shared subword intersection) but add per-step CPU work and depress acceptance. The practical answer is usually to stop looking for a second model at all.
Medusa: extra heads instead of a second model
Self-speculation removes the drafter by making the target model predict several steps ahead of itself. Medusa is the simplest form: bolt k extra lightweight decoding heads onto the target’s final hidden state, where head i is trained to predict the token at position t+i+1. Each head is a residual block plus a vocabulary-sized projection — a few hundred million parameters total, trainable in hours on a frozen backbone.
The advantages are structural rather than statistical: no second model to load or schedule, no tokenizer mismatch by construction, and the heads read the target’s own hidden state, so their guesses are conditioned on exactly what the target knows. The catch is that head i predicts position t+i+1 without seeing what was sampled at t+1…t+i, so accuracy decays sharply with i. Medusa compensates by taking the top few candidates per head and verifying them as a tree, and by relaxing strict rejection sampling into a typical acceptance rule — which is fast, but is no longer distribution-exact above temperature zero.
EAGLE, Lookahead, and drafting with no model at all
EAGLE fixes Medusa’s independence problem by moving autoregression down a level. Instead of predicting tokens in parallel, it autoregressively predicts the target’s hidden-state feature vector one step ahead, using a single transformer layer fed the pair (feature_t, embedding of the token sampled at t+1), then reuses the target’s frozen LM head to turn each predicted feature into a token distribution. Feature space is smoother and lower-entropy than token space, so a one-layer drafter gets remarkably long accepted runs. Later versions expand the candidate tree adaptively, where the drafter’s confidence is high.
At the other extreme, Lookahead decoding needs no trained component: it runs Jacobi-style parallel updates, harvests the n-grams they produce into a pool, and verifies pool hits alongside the normal step. Simpler still, prompt-lookup drafting matches the last few generated tokens against the prompt and copies whatever followed. That is a string search — zero parameters, zero bandwidth — and it is devastatingly effective on summarization, RAG, code editing, and reformatting, where most output is copied input.
Tree drafting and the tree attention mask
A linear draft of K tokens dies at the first rejection. A tree hedges: propose several alternatives at each depth and verify all root-to-leaf paths in the same forward pass, then accept the longest path that survives. The mechanism is a custom attention mask. Flatten the tree’s M nodes into one sequence, set each node’s position id to its depth, and allow node i to attend only to itself and its ancestors:
tree: r → {A, B}, A → {C}
flatten: [r, A, B, C] pos_ids = depth = [0, 1, 1, 2]
r A B C
r 1 . . .
A 1 1 . .
B 1 . 1 .
C 1 1 . 1B never sees A, so each path is scored as if it were the only continuation — one weight sweep, many hypotheses. Trees are not free: attention and the FFN both scale with M, so a wide tree pushes the verify pass off the bandwidth plateau. Use a small calibrated sparse tree, not a full b^d expansion.
KV-cache surgery when a draft is rejected
Verification writes keys and values for all M speculative positions into the target’s KV cache, but only the accepted path is real. Everything else must disappear before the next step, or the model attends to tokens it never emitted — a corruption bug that surfaces as subtly incoherent text rather than a crash.
For a linear draft the fix is trivial: the cache is append-only, so truncating the length counter to prefix + a after accepting a tokens discards the rest. Tree drafting is harder, because the accepted path’s entries are scattered across the M flattened slots. Implementations gather the surviving rows per layer and compact them into contiguous positions; under paged attention this is a block-table rewrite plus a small slot copy. Two related costs: the drafter’s own cache needs the same rollback, and every in-flight sequence must reserve M slots it will mostly throw away, which lowers how many sequences fit in memory at once.
Why continuous batching eats the speedup
This is the result that surprises people who benchmarked on a laptop and then deployed. Speculation exploits idle arithmetic units — but continuous batching already does that. At batch size B, one weight sweep serves B tokens, so arithmetic intensity is already B× higher; add speculation and each step processes B · M token-positions.
Once B · M passes the hardware’s ops-to-bytes ratio, the target model is compute-bound and every speculative position costs real time in proportion. You are then paying for B · (M − a) discarded token-computations per step with no bandwidth slack left to hide them. The speedup decays from roughly E[accepted] at B = 1 toward < 1 — a net regression — under load. Speculation is a latency optimization, not a throughput one, which is why serving stacks gate it dynamically: shrink K or disable it once the running batch exceeds a threshold, and re-enable when the queue drains.
Speculation on a CPU
The CPU-SLM case is the friendliest one there is, because it sits permanently in the regime where speculation wins: batch size 1, one interactive user, and a bandwidth wall far tighter than a GPU’s. Dual-channel DDR5-5600 delivers roughly 90 GB/s, so a 7B model quantized to about 4 GB has a hard floor near 4/90 s ≈ 44 ms per token — roughly 22 tok/s no matter how many cores you own.
Meanwhile AVX-512 or AMX gives a desktop CPU an ops-to-bytes ratio in the low tens, versus several hundred on a datacenter GPU. The free-token budget is real but small, so the profitable tree is narrow — think M of 4 to 8, not 64. A 0.5B drafter at ~350 MB costs about 4 ms a step, affordable against a 44 ms verify. And because memory is the scarce resource here, prompt-lookup drafting is disproportionately attractive: no weights, no bandwidth, no second cache.
Failure modes, and when to turn it off
Speculation has a genuine downside case, not merely a neutral one. Every rejected token is wasted draft latency, so when acceptance is low the whole cycle is slower than plain decoding. That happens with high-temperature or open-ended creative generation, with an out-of-domain drafter (a chat-tuned drafter on a code target), and with any lossy self-speculation head whose training data drifted from the deployment traffic.
Three subtler traps. Long context: speculation amortizes the weight sweep, not attention over the KV cache — at 128k tokens the attention term dominates and scales with M, so the win shrinks. Memory: reserved speculative slots reduce concurrency and can trigger preemption. Reproducibility: even with exact rejection sampling, verifying M positions changes kernel shapes, and the floating-point differences can flip a near-tie argmax, so greedy output may not be bit-identical to the non-speculative run. Measure acceptance in production; below roughly 50–60%, switch it off.