Speculative decoding is usually explained as a sampling trick: a small draft model proposes tokens, a large target model verifies them, and a rejection rule keeps the output distribution exactly right. That is the algorithm. This article is about the machine. On a GPU the technique is something narrower — a way of spending arithmetic you were already throwing away, because autoregressive decode leaves a datacenter accelerator starved for work between memory transfers. Read it that way and the practical questions answer themselves: how many tokens to draft, what it costs in memory, why tree verification is expensive, and why it stops paying the moment your server gets busy.
Decode is a bandwidth problem, not a math problem
Autoregressive generation produces one token per forward pass, and each of those passes has to read every weight in the model out of HBM. For a model of N parameters in 16-bit, that is roughly 2N bytes of memory traffic to perform roughly 2N floating-point operations — on the order of one FLOP per byte moved.
Modern accelerators are built for the opposite ratio. Divide a datacenter part’s peak tensor-core throughput by its HBM bandwidth and you land in the hundreds of FLOPs per byte before the math units saturate. Decode therefore sits far to the left of the roofline ridge point: step duration is set almost entirely by how long it takes to stream the weights, and the SMs spend that time waiting on loads with the tensor cores idle. Kernel tuning cannot remove the bottleneck, because the weights genuinely have to cross the bus. The only lever left is to extract more tokens per crossing.
Arithmetic intensity: what changes when you verify k tokens
That is precisely what speculation buys. Instead of one forward pass yielding one token, the target model takes k drafted tokens and evaluates them in a single pass, emitting k+1 output distributions. The weight traffic is unchanged — still one sweep of 2N bytes — while the arithmetic multiplies by k+1.
In roofline terms the operation moves right along the intensity axis, from about 1 FLOP/byte toward k+1, and achieved throughput climbs the memory-bound diagonal with it. The step gets longer, but nowhere near k times longer, because the extra math hides under memory traffic that was happening anyway. That is the entire GPU-side case for speculative decoding: it converts idle bandwidth-bound headroom into candidate tokens. Note the boundary condition baked into that sentence — it works only while the pass is still memory-bound. Push intensity past the ridge point and every extra verified position costs real time.
From GEMV to a skinny GEMM — feeding the tensor cores
The shape change is where the idle silicon lives. Serving one sequence one token at a time makes every projection a matrix-vector product: the activation is a single row, M = 1. A GEMV has no reuse to exploit — each weight element is loaded and used once — so it cannot fill a tensor-core tile.
Tensor-core MMA instructions consume fixed tiles whose M dimension is 8, 16 or 64 depending on the generation. A one-row activation occupies one row of that tile; the rest is padding you pay for regardless. Verifying k drafted positions makes M = k: a short, wide GEMM, still memory-bound, but now using tile rows that were previously wasted. That is tile quantization working in your favour, and it explains why raising k from 1 to a handful is nearly free on the target side.
Where the time goes in one speculative iteration
Budget an iteration by which weights move rather than by which decision is made. Two costs occur. First, k small forward passes stream the draft model’s weights, once each, sequentially. Second, one forward pass streams the target model’s weights — the expensive sweep that dominates — covering all k+1 positions at once.
The accept/reject rule that follows is elementwise work on logits, negligible in the timeline. The engineering question is how many committed tokens you get per target sweep versus the draft-side time and memory spent earning them.
The draft step has a latency floor
The seductive assumption is that a draft model 1/70th the size runs 70× faster per step. It does not, and understanding why bounds useful values of k more sharply than acceptance rate does.
The draft model is bandwidth-bound too, so its step time scales with its weight bytes — but only down to a floor. Costs that do not shrink with parameter count keep accumulating: the same per-layer kernel launches, the same host-side scheduling and synchronization, sampling and logit post-processing, and reads of a KV cache whose sequence length matches the target’s. Below a certain size the model stops being bandwidth-bound and becomes latency-bound, dominated by many small kernels that under-occupy a large GPU. Because those k draft steps are strictly sequential, their floors add up. Each extra drafted token costs a full draft step at floor price, while one more verified position costs the target almost nothing.
Launch counts and the shapes that defeat graph capture
The sequential draft chain also multiplies launch overhead. A decode iteration is short GPU work to begin with, so the CPU-side cost of enqueuing kernels is already a visible fraction of it, and k extra passes multiply the launches issued before any token is committed.
Capturing the decode step as a replayable graph is the usual remedy, and speculation is unusually hostile to it. A captured graph is fixed in shape; a speculative iteration is not. The number of draft steps taken can be cut short, the accepted count varies from 0 to k, and any tree of candidates changes node count between iterations. Implementations respond by padding to a fixed maximum k and masking the unused slots, or by keeping a small library of captured shapes. Padding is not free — you execute draft steps whose tokens may never be used.
The KV cache you spend on speculation
Speculation costs HBM, and the bill arrives somewhere easy to miss. The draft model carries its own KV cache for every live sequence, with its own layer count and head dimensions, growing with context length and batch size — capacity the target model’s cache does not get.
Verification adds a transient claim: k+1 speculative positions per sequence must be written before anyone knows how many survive. A paged allocator either reserves those blocks in advance or trims the tail of the last block after rejection; either way the scheduler holds per-sequence headroom that frequently never becomes committed context. The second-order effect is what matters. Less free KV memory means fewer concurrent sequences, and on a throughput-oriented server the maximum batch size is the throughput number — so speculation can cost throughput before executing a single extra FLOP.
Tree verification and the shape of the attention mask
A linear draft of k tokens verifies against a plain causal mask over a k-length suffix — exactly the shape fused attention kernels already handle. Submitting a tree of alternative continuations breaks that assumption, and the cost lands in the kernel.
In a tree, each candidate node may attend only to its own ancestors, and that relation is not causal in position order: node 7 may be a sibling of node 4 rather than its descendant, so visibility cannot be derived from indices. The kernel needs an explicit per-node mask — a materialized boolean block or a packed bitmask — and the fused fast path must accept it, meaning a kernel variant beyond the usual causal and full cases. You lose the skip-the-upper-triangle shortcut, pay to build and read the mask, and the query length becomes the node count rather than the accepted-token count. Trees buy coverage sublinearly while adding attention work linearly in nodes.
Batch size is the variable that decides everything
Every argument above assumed the target pass was memory-bound — true at batch size 1, progressively less true as the server fills up. At batch B the decode GEMM already has M = B: one weight sweep is amortized across B sequences, so arithmetic intensity is roughly B FLOPs per byte before any speculation.
Once B is large enough to cross the ridge point, the pass is compute-bound and the free lunch is over: verifying k+1 positions per sequence multiplies real execution time, not just hidden math. Worse, speculation is intrinsically wasteful — every rejected token is arithmetic performed and discarded, and a compute-bound GPU pays full price for it. Stated plainly, speculation and batching are two ways of spending the same slack. Batching is the more efficient one for throughput, so when both are available, batching wins and speculation becomes overhead.
Turning it off — speculation as a scheduling decision
Speculation is therefore a latency optimization with a throughput cost, and it belongs in the scheduler rather than a static config file. The gates worth wiring up are running batch size or queue depth, a smoothed measurement of recent acceptance, and the SLO you are actually serving — inter-token latency for chat, tokens per GPU-second for offline work.
A robust policy falls out: speculate aggressively when the machine is quiet and the batch is small, shrink k as the batch grows, and disable it entirely under sustained load. Continuous-batching schedulers complicate the mechanics, since a running batch may mix speculative and ordinary sequences with different step shapes — another reason to switch at coarse thresholds rather than per token. Validate end to end, because the KV headroom cost shows up as a lower achievable batch size long before it appears in any kernel profile.