Why architecture matters here
Speculative decoding architecture matters because it delivers real speedup without quality loss. Small draft models are 10-30x faster than target; when acceptance rate is 60-80%, effective throughput is 2-3x target-only.
Cost is the extra draft model (small). Well worth it when acceptance is high.
Reliability comes from correctness proof: mathematical guarantee the output distribution matches target-only sampling.
The architecture: every step explained
Walk the diagram top to bottom.
Draft Model. Small (say, 1B); fast on same hardware. Ideally same tokenizer + trained on similar data.
Target Model. Large (say, 70B); the real model users want.
Speculative Loop. Iterative: draft proposes K tokens; target verifies; accept + generate one more from target if all accepted, or accept prefix + resample at first rejection.
Draft K tokens. Draft runs K forward passes autoregressively, cheap.
Verify in single target forward. Target processes prompt + K drafts in one forward, producing K+1 output distributions.
Accept prefix. For each position, compare draft's chosen token to target's distribution; accept with probability min(1, p_target / p_draft) per rejection sampling.
Reject at mismatch. First rejection: resample from adjusted target distribution.
Correctness proof. Rejection sampling ensures accepted-or-resampled token has target's distribution. Provably identical to target-only sampling.
Acceptance rate. 60-80% typical; depends on draft quality.
Variants. Medusa: multi-head prediction from target itself. EAGLE: learn draft from target's features. Self-speculative: use target's earlier layers as draft.
End-to-end iteration flow
Trace an iteration. Prompt processed by target; first token generated. Now speculative loop starts.
K = 4. Draft (small model) proposes 4 tokens autoregressively.
Target processes all 4 in single forward. Outputs 5 distributions (one per position + one extra).
Position 1: draft chose "the" with p_draft = 0.3; target's p_target for "the" = 0.4. Accept (min(1, 0.4/0.3) = 1).
Position 2: draft "quick" p_draft = 0.2; target p_target = 0.1. Accept with probability 0.5. Random gives 0.7 → reject.
Accepted prefix: "the". Resample position 2 from target's adjusted distribution. Get "brown".
3 tokens generated in one target forward pass. Speedup: 3x (vs one at a time).
Repeat. Over many iterations, average of ~2.5 tokens accepted per target step. Effective 2.5x throughput.