EAGLE makes token generation faster by drafting cheaply and then verifying in a single pass — the same draft-then-verify idea behind all speculative decoding. What makes it fast is what it drafts. Vanilla speculative decoding runs a separate small model to guess tokens; Medusa bolts extra heads onto the target to guess several tokens in parallel. EAGLE instead drafts at the feature level — the second-to-top hidden state that the model’s LM head consumes — using one tiny autoregressive head that predicts the next feature, conditioned on the previously sampled token to resolve the ambiguity that sampling introduces, and then reuses the target’s own LM head to turn that feature back into a token. That single design choice buys a markedly higher acceptance rate, and acceptance rate is the whole speedup — as the feature-drafting math, a worked speedup example, and EAGLE-2’s dynamic draft trees below make concrete.
Decode is memory-bound — why draft-then-verify wins
Autoregressive decode generates one token per forward pass, and each pass must stream the entire weight matrix from memory to compute a single new position. The arithmetic is trivial; the memory traffic is not. So decode is memory-bandwidth-bound: the hardware’s compute units sit mostly idle, waiting on weights. A forward pass that scores one position and one that scores a dozen cost almost the same wall-clock time, because both are gated by loading the weights, not by the flops.
Speculative decoding exploits exactly this slack. Cheaply guess the next γ tokens with some fast drafter, then run the expensive target model once to score all γ+1 candidate positions in parallel. Verification is lossless: a rejection-sampling check accepts the longest correct prefix and preserves the target’s exact output distribution. If the guesses are good you collect several tokens for the price of one weight-load. The whole game is a drafter that is both cheap and accurate.
Vanilla speculative decoding: a separate draft model
The original recipe uses a second, smaller model q to draft tokens autoregressively, then the target p verifies them in one pass. Each drafted token x is accepted with probability min(1, p(x)/q(x)); on the first rejection you resample that position from the adjusted distribution (p − q)_+ and stop. This is provably equivalent to sampling from p directly — no quality loss.
Let α be the average per-token acceptance probability. The expected number of tokens produced per verification pass is a geometric sum:
E = (1 − α^(γ+1)) / (1 − α)The +1 is the free bonus token the verification pass always yields. The catch is practical: you must find or train a separate draft model whose distribution is close enough to the target to earn a high α, keep it aligned across fine-tunes, and pay to deploy two models. A good small drafter for a given target is genuinely hard to come by.
Medusa: independent token heads
Medusa removes the second model. It freezes the target and attaches K extra lightweight heads on top of the last hidden state h_t; head k predicts the token at position t+k directly, and all heads fire in parallel from the same h_t. No autoregression, no separate network — one forward pass emits several draft tokens at once.
The weakness is structural. Head 2 predicts the token two steps out without knowing what head 1 actually produced. It is modelling the marginal p(t+2) rather than the conditional p(t+2 | t+1), yet real text is highly conditional — the next word depends on the word just chosen. Because each head ignores its predecessors’ realized outputs, joint accuracy decays quickly with distance and acceptance falls off. Medusa recovers some of this by verifying a tree of candidate combinations, but the root cause — independent, non-autoregressive heads — remains. That is the gap EAGLE closes.
EAGLE’s move: draft at the feature level
EAGLE’s insight is that tokens are the wrong thing to draft. A token is a discrete, high-entropy sample; the vector that produces it is smoother and more regular. That vector is the feature f — the second-to-top hidden state, the output of the final decoder layer that the LM head consumes to form the next-token distribution. Feature sequences evolve far more predictably than token sequences, so extrapolating one step ahead in feature space is an easier learning problem than guessing the next token outright.
So EAGLE runs a single small autoregression head — one extra decoder layer — over the sequence of features, predicting the next feature f′ from the current one. It then applies the target’s own, unchanged LM head to f′ to get a token distribution and samples from it. No second model, no independent heads: just cheap autoregression in feature space, decoded through the very head the target already uses. Reusing that exact LM head is a big part of why the drafts stay close to the target’s distribution.
The uncertainty problem, and the token that fixes it
There is a subtlety that would sink naive feature extrapolation. The map from one feature to the next is not deterministic. A feature f yields a distribution over next tokens; which token is actually sampled is part of the state, and it changes the next feature. So f alone underdetermines f′ — the same feature can lead to different continuations depending on the draw. This is the exact ambiguity Medusa’s independent heads swallow.
EAGLE resolves it by feeding the head the token that was actually sampled. One draft step holds the pair (f, t) and advances it:
f′ = AR_Head( f , e(t) ) # e(t): embedding of the sampled token
p = softmax( LM_head(f′) ) # target’s own head, unchanged
t′ ~ p # sample the next token
(f, t) ← (f′, t′) # advance one step, repeat γ timesThe head’s input is the feature concatenated with the shifted token embedding, so it always knows which branch of the sampling was taken. That single conditioning term collapses the uncertainty that non-autoregressive heads cannot.
Why feature drafting lands a higher acceptance rate
Two things push EAGLE’s per-token distribution close to the target’s, which is what raises α. First, it decodes drafted features through the target’s exact LM head, so the final feature→token mapping is identical rather than approximated by a different network. Second, it is genuinely autoregressive and conditions on the realized token, so it models p(t+2 | t+1) instead of the marginal. A separate draft model suffers a distribution mismatch; Medusa’s heads suffer a dependency mismatch; EAGLE has neither.
The head is trained cheaply on the target’s own recorded features with a two-part loss: a smooth-L1 regression term that makes the predicted feature f′ track the true next feature, plus a cross-entropy term on the decoded token distribution. Aligning both the vector and the distribution lets one small layer imitate the target well enough to earn acceptance rates well above token-level or independent-head drafting — and, as the next section shows, acceptance rate is where all the speedup lives.
The speedup math, worked
Recall that one target forward pass costs roughly the same whether it scores 1 or γ+1 positions, because it is memory-bound. So each verification emits E accepted tokens (from the formula above) at the price of one target pass plus γ cheap draft steps. With c the ratio of one draft step to one target pass, the wall-clock speedup over plain decode is:
speedup ≈ E / (1 + γ·c) , where E = (1 − α^(γ+1))/(1 − α)Hold γ = 4 and c ≈ 0.1 and vary only α. A separate small drafter at α = 0.7: E = (1 − 0.7^5)/0.3 ≈ 2.77, speedup ≈ 2.77/1.4 ≈ 1.98×. Lift acceptance to α = 0.8: E ≈ 3.36, speedup ≈ 2.40×. Push it to α = 0.9: E ≈ 4.10, speedup ≈ 2.93×. Same γ, same draft cost — the entire gain comes from acceptance rate, which is exactly what feature-level drafting buys. And EAGLE’s drafter (one layer plus the shared LM head) is cheaper still than a separate model, shrinking c and nudging the number up further.
EAGLE-2: dynamic draft trees
Both EAGLE-1 and Medusa verify a static draft tree — the same fixed shape of candidate continuations every step. That wastes budget, because acceptance is context-dependent. Sometimes the top draft is nearly certain and you should extend it deep with few branches; sometimes the next token is a coin flip and you should branch wide but shallow. A fixed tree can do neither well.
EAGLE-2 makes the tree dynamic. Crucially, the draft head’s own output confidence turns out to be a good proxy for the true acceptance probability, so EAGLE-2 uses those draft probabilities to grow the tree greedily: expand the nodes with the highest cumulative draft probability, prune low-probability paths, and re-rank which candidates to keep. The number of tree nodes verified per target pass — the compute budget — is unchanged; it is simply allocated where it pays off. The result is a longer expected accepted length per pass at no extra target cost. A tree attention mask lets every candidate in the tree be scored in that single verification pass.
Practical notes and pitfalls
It is lossless. Verification is the standard speculative accept/resample step, so EAGLE’s output distribution is identical to the target’s — it is an acceleration, not an approximation, and does not change what the model would have said. Feature drift limits depth. Errors in predicted features compound as you draft further ahead, so acceptance decays with draft depth; there is an optimal γ (or tree size) beyond which the extra draft cost outweighs the extra accepted tokens. The head is target-specific. It trains on one model’s features at a given precision and must be retrained per target.
For CPU-hosted small models the memory-bound premise is even stronger, so a cheap accurate drafter is attractive — but on CPU the draft head adds real compute, so c is less favorable than on a bandwidth-starved GPU; keep γ modest. Batching also cuts both ways: at large batch size the target pass turns compute-bound rather than memory-bound, the per-pass slack speculative decoding harvests shrinks, and the speedup with it.
α up, and since expected accepted length E = (1 − α^(γ+1))/(1 − α) — and thus the speedup — rises steeply with α, that is the win. EAGLE-2 goes further, spending a fixed verification budget on a dynamic draft tree shaped by the drafter’s own confidence. In speculative decoding, accuracy of the draft beats raw cheapness — and features are the more predictable thing to draft.