EAGLE is the strongest of the self-drafting speculative-decoding families, and its companion piece already covers why drafting at the feature level beats a separate token model or Medusa’s independent heads, along with the geometric-sum speedup. This deep dive assumes that and goes under the hood. We write out the exact autoregressive recurrence the draft head runs, count what is actually in that head, and then follow the line from EAGLE-1 to EAGLE-2’s dynamic draft trees — the expand-and-rerank algorithm, the tree-attention mask, and the acceptance-length math a tree changes — to EAGLE-3, whose training-time test and multi-layer feature fusion drop the one constraint that capped the earlier versions and let acceptance keep climbing with more data. Throughout, the question is the same: how do you maximize the expected accepted length per verification pass, because that number, and nothing else, is the speedup.
The recurrence, written out
EAGLE’s draft head is a genuine autoregressive model, so it helps to see its one-step update explicitly. Let f_i be the target’s second-to-top hidden state at position i (its feature, shape [d]), and let t_{i+1} be the token that was sampled from f_i through the LM head. The head forms an input by concatenating that feature with the embedding of the just-sampled token, projects it back to width d, and runs one decoder layer:
x_i = concat( f_i , emb(t_{i+1}) ) # [2d]
h_i = fc( x_i ) # [2d] -> [d]
f'_{i+1} = DecoderLayer( h_i , cache ) # [d] predicted next feature
t_{i+2} ~ softmax( LM_head( f'_{i+1} ) ) # decode with the TARGET's headThe predicted feature f′_{i+1} then plays the role of f in the next step, and so the head rolls forward γ times to draft a chain (or, below, a tree). Two things make this cheap and honest: only fc and one decoder layer are new, and the token is produced by the target’s own frozen LM head, so the drafted distribution is anchored to the real model.
What is actually in the head
It is worth being concrete about how little the draft head contains, because that is where the ‘draft cost’ term c in the speedup formula comes from. The head is: one linear layer fc of shape [2d, d], a single transformer decoder layer (self-attention plus MLP, the same width d as the target), and shared copies of the target’s token embedding and LM head — shared, so they add no new parameters and no new memory traffic beyond what the target already pays.
For a model with hidden size d and vocabulary held in the shared head, the trainable additions are roughly 2d^2 (the fc projection) plus one decoder layer’s ≈12d^2 — on the order of a single extra layer, a percent or two of a many-layer target. That is the point: the drafter is small enough that reading its weights and running γ steps costs a fraction of one target forward pass, keeping c small so almost all of the accepted-length gain survives as real wall-clock speedup.
Exposure bias: the multi-step training gap
An autoregressive drafter has a subtle training problem the base article does not dwell on. During training it is natural to feed the head the target’s true recorded features f_i and ask it to predict f_{i+1}. But at inference, after the first step, the head consumes its own predicted feature f′, not the true one. This is classic exposure bias: the deeper into a draft you go, the more the input distribution drifts away from anything the head saw in training, and acceptance decays with draft depth.
EAGLE-1 mitigates this by adding noise to the input features during training so the head learns to be robust to small feature errors, and by keeping drafts relatively shallow. It is a patch, not a cure — the mismatch between teacher-forced training and free-running inference is exactly the wall EAGLE-3 later attacks head-on with its training-time test. For now, note that feature drift is the structural reason there is an optimal draft depth rather than ‘deeper is always better.’
EAGLE-2: expand and rerank
EAGLE-2 keeps the same draft head and makes the draft shape adaptive through two phases per step. In the expand phase it grows the tree greedily: each node carries a value equal to the product of the draft confidences along the path from the root to it — an estimate of the probability that the whole prefix is accepted — and only the highest-value leaves are expanded by running the draft head to produce their top-k children. Low-value paths are never grown, so budget is not wasted on branches unlikely to be accepted.
In the rerank phase, once expansion is done, EAGLE-2 pools all nodes generated and keeps the top m by value to actually verify. Because a node’s value already accounts for its ancestors, this naturally favors a few deep, confident continuations plus some shallow hedges. The number of nodes verified — the target-side compute budget m — is fixed; EAGLE-2 only changes which m nodes, spending the same budget where it converts to accepted tokens.
Why draft confidence is a usable acceptance proxy
The whole scheme rests on one empirical fact: the draft head’s own output probability for a token is a good proxy for the probability that verification will accept that token. That is not obvious a priori, but it follows from what the head was trained to do — imitate the target’s next-token distribution. When the head is confident, it is usually confident because the target is too, so the standard speculative accept test (accept with probability min(1, p_target/p_draft)) passes most of the time.
This is what lets EAGLE-2 rank nodes without ever consulting the target during drafting. A path’s value ∏_j p_draft(t_j) approximates P(all tokens on the path accepted), so maximizing total value across the kept nodes approximately maximizes expected accepted length — the quantity we care about — using only cheap draft-side information. The correlation is imperfect, which is why it is a heuristic and not a proof, but it is strong enough that dynamic trees reliably beat static ones of the same size.
Acceptance length under a tree
For a single drafted chain, expected accepted length is the geometric sum the base article derives, E = (1 − α^{γ+1})/(1 − α). A tree changes the accounting because several candidate tokens are offered at each depth, and verification accepts the deepest path whose every token passes. If depth ℓ offers b independent candidates each accepted with probability α, the chance at least one is accepted rises to 1 − (1 − α)^b.
Expected accepted length is then Σ_{ℓ≥1} P(reach depth ℓ), and widening the tree lifts each term. A worked feel: at α = 0.7 a single chain reaches depth 2 with probability 0.49; offering b = 3 candidates there lifts it to 1 − 0.3^3 ≈ 0.97. The tree is buying accepted length by hedging against a single unlucky rejection — and, crucially, it does so under a fixed verification budget, which is exactly why EAGLE-2’s reallocation of that budget, rather than growth of it, is the win.
Tree attention in one verification pass
The reason a whole tree of candidates can be verified for the price of one target forward pass is a custom attention mask. All tree nodes are flattened into a single sequence, but the mask lets each node attend only to its ancestors in the tree, never to siblings or cousins. Position ids are set to the node’s depth, so every candidate sees a causally consistent prefix even though unrelated branches share the same physical positions in the batch.
The effect is that one pass over m tree tokens produces, for every node, the target’s next-token distribution conditioned on exactly that node’s path — all the numbers verification needs. Cost scales with the tree size m, not with a separate pass per candidate, and because decode is memory-bound the extra tree tokens are near-free arithmetic on weights already loaded. The mask is the mechanical trick that turns ‘verify many guesses’ from m forward passes into one wider one.
EAGLE-3: the constraint that capped the earlier versions
EAGLE-1 and EAGLE-2 both require the draft head to predict a feature — the target’s second-to-top hidden state — and train it with a regression loss that pulls f′ toward the true f. That feature-alignment objective is what makes reusing the LM head valid, but it is also a ceiling: the head is spending capacity matching a high-dimensional vector exactly, and once it does that about as well as a one-layer model can, adding more training data stops helping. Acceptance plateaus.
EAGLE-3’s central observation is that predicting the feature was never the goal — predicting the accepted token was. So it drops the feature-regression loss entirely and trains the head to predict tokens directly. Removing that constraint is what makes the head’s quality, and therefore the acceptance rate, scale with data again rather than saturating — the property the paper’s title (‘scaling up… via training-time test’) advertises.
The training-time test
Dropping feature regression reopens the exposure-bias problem, because now nothing forces the drafted features to stay in-distribution as the head free-runs. EAGLE-3’s fix is the training-time test: instead of training each step against ground-truth features (teacher forcing), it simulates the multi-step draft during training — the head consumes its own predicted intermediate outputs across several steps, exactly as it will at inference, and is supervised on the tokens at every simulated step.
This closes the train/inference gap directly: the head learns to be accurate on the inputs it will actually see, including its own accumulated errors, rather than only on clean recorded features. It is more expensive to train because each example now unrolls several steps, but it removes the drift-driven acceptance decay that forced EAGLE-1 to stay shallow and to inject noise as a crutch. The payoff is that deeper drafts stay accurate, so the optimal γ moves out and expected accepted length per pass goes up.
Multi-layer feature fusion
EAGLE-3 also changes what the head reads. EAGLE-1/2 feed only the single second-to-top feature; EAGLE-3 fuses hidden states from low, middle, and high layers of the target — roughly g = concat(h_low, h_mid, h_high) — giving the drafter a richer view than any one layer provides. Low layers carry more lexical/positional signal, high layers more semantic and next-token commitment; combining them lets one small head condition on both.
Concretely the fused vector is projected down and used in place of the lone f in the recurrence. This is only affordable because the target already computed all those hidden states in its verification pass — EAGLE-3 reads activations that are otherwise thrown away, so fusion adds capture and a small projection but no extra target compute. Together with the training-time test, fusion is why EAGLE-3 reports roughly a further 1.3–1.4× over EAGLE-2 and total speedups pushing past 5× on friendly workloads, while remaining exactly lossless — verification is unchanged, so the output distribution is still the target’s.
What it means at batch 1 on a CPU
Speculative decoding’s premise is that single-stream decode is memory-bandwidth bound: each token drags the whole weight set through the memory hierarchy while the ALUs idle. That description fits a CPU running a small model at batch 1 almost perfectly, which is why EAGLE is attractive there — it trades the CPU’s spare arithmetic (a tree of candidates, one wide pass) for fewer of the scarce weight-streaming passes that actually gate throughput.
Two practical notes follow. First, the draft head is small enough that its weights can stay resident in cache between steps, so its per-step cost really is close to the c the math assumes. Second, the tree’s KV cache and the wider verification pass cost memory and a little bandwidth of their own, so on a bandwidth-starved machine the sweet-spot tree is smaller than on a GPU — tune m and γ for your bandwidth-to-FLOP ratio. The KPI to optimize is unchanged from the base article: mean accepted tokens per verification pass, measured on your real prompts.
Pitfalls specific to the deep version
Three traps show up once you push EAGLE hard. Per-target, per-precision retraining. The head is trained on one model’s features at one quantization; swap the target, or quantize it differently, and the features shift, so a head trained on the fp16 model can lose acceptance on the int4 one. Retrain against the deployment precision. Tree size is not free. Bigger trees raise accepted length with diminishing returns while their verification cost and KV footprint grow linearly; past the optimum, throughput falls even as accepted length inches up.
Acceptance is workload-dependent. Predictable, templated, or code-like text drafts far better than open-ended creative prose, so a speedup measured on one distribution will not transfer to another. Always report accepted length on representative traffic, and re-tune when the workload changes. None of these threaten correctness — EAGLE stays lossless throughout — but each quietly erodes the speedup if ignored.
fc projection plus a single decoder layer that rolls a feature forward autoregressively, decoding through the target’s frozen LM head to stay honest — but rolling forward invites exposure bias, so acceptance decays with depth. EAGLE-2 spends a fixed verification budget better by growing a dynamic tree with expand-and-rerank, ranking nodes by the product of draft confidences (a proxy for acceptance) and verifying the whole tree in one pass via an ancestor-only attention mask. EAGLE-3 removes the feature-prediction constraint that capped the earlier versions: a training-time test trains the head on its own multi-step outputs to kill the train/inference gap, and multi-layer feature fusion feeds it low, middle, and high activations the target already computed. The result scales with data and is still exactly lossless. On a batch-1 CPU, all of this trades abundant arithmetic for scarce weight-streaming passes — so tune the tree to your bandwidth, retrain per precision, and optimize the one number that matters: mean accepted tokens per verification pass.