A Field Guide to Model-Based Speculative Drafting

In How Speculative Decoding Works (and When It Doesn't), I covered the basic algorithm, why exact verification preserves the target distribution, and when speculation stops helping. A quick recap: a cheap drafter proposes several tokens, and the target scores the whole draft in one forward pass. Verification accepts a consecutive prefix. At the first rejection, it samples one correction token from the residual distribution and discards the remaining draft tokens. If the full draft is accepted, the target contributes one bonus token. So in this post we’ll take that machinery and look at a different question: what should the drafter actually be?

With exact speculative-sampling verification ^[1], the output distribution is unchanged even when the drafter is inaccurate. The model-based approaches in this post all rely on that property. But just because the drafter can be sloppy doesn’t mean it’s free. The draft and verification work is paid on every cycle. Low acceptance means fewer output tokens are produced per target pass, which can erase the speedup. So the design of your draft model is very important.

And that’s exactly why these methods differ so much. Each draft architecture makes a different tradeoff between $\tau$ and draft cost. Here, $\tau$ is the average number of output tokens produced per target verification pass. A higher $\tau$ amortizes each target pass over more output tokens. A cheaper drafter reduces the cost paid per cycle. One note before we dive in: we’re only going to discuss drafters with trained weights. I’ll cover training-free approaches such as $n-$gram drafting, prompt lookup, and lookahead decoding in a separate post. They have different tradeoff axes and can sometimes even be used in conjunction with model-based drafters.

The reference drafter

Every approach we will discuss works around a limitation of the classic target-draft setup. The classic setup uses a separate, smaller model that drafts tokens autoregressively before each target verification pass. It works, but it creates several engineering constraints.

First, you need a cheap drafter with a compatible token vocabulary. The drafter must be much cheaper per generated token than the target. Parameter count alone does not determine that cost. Plenty of model families don’t ship anything like that. Without a compatible off-the-shelf drafter, you may need to train or adapt one. A model from another family can have a less aligned output distribution, which can lower acceptance. Standard speculative sampling assumes a shared token vocabulary. Different tokenizers require an explicit alignment method, so they are not a drop-in pairing.

Second, it is an entire additional model to serve. It needs separate weights, its own KV cache, and separate scheduling. If the target already fills the node, the drafter must share reduced resources, run elsewhere, or use another placement strategy. Separate placement adds communication and synchronization on every speculation cycle. Co-location constrains the memory and parallelism layouts available to both models.

Finally, the drafter and target maintain separate hidden states and KV caches for the accepted prefix. The two models may encode that prefix differently, but both representations are current. Every verification step, the target computes its own internal representation of the drafted tokens and appends them to its KV cache. A standalone drafter does not reuse those target states in the next speculation cycle. In other words, the most up-to-date representation of the context always lives inside the target model, but the drafter never gets to use it.

In practice, a well-matched independent drafter often delivers around $1.5-2\times$ wall-clock speedup, although the exact number depends heavily on the workload, hardware, draft-to-target compute ratio, and acceptance rate ^[1]. Almost every model-based drafting method we’ll discuss can be viewed as an attempt to address one or more of these constraints: either by getting rid of the separate model entirely, sharing computation with the target, or reusing the target’s own internal representations.

Medusa 1 & 2

Most of the problems with the classic target/draft setup trace back to the same root cause: the drafter is a separate model. Medusa’s answer is to stop using a separate model. Instead of a sibling model, you attach a handful of small prediction heads directly onto the target’s last hidden layer. The original LM head predicts the next token $t+1$. The first Medusa head predicts $t+2$, the second predicts $t+3$, and so on. All heads read the same final hidden state. The original paper used five additional heads, giving predictions out to six positions in total. Each head is a small feed-forward block with a SiLU activation and a residual connection. Its output projection is initialized from the model’s LM head. Compared with a separate Transformer drafter, the added draft computation is small ^[2].

Notice that it solves all three problems from the last section. There is no separate sibling model to source or distill. The added heads are a small fraction of the target model. There is no separate draft model, draft KV cache, or inter-model communication path. The heads run from the target’s hidden state. The heads draft directly from the target’s current hidden state, so they do not maintain a separate draft representation.

The catch is that these heads don’t relate to each other. That’s Medusa’s core limitation: every head sees the same $h_t$. So head $3$ is guessing the token three positions out without knowing what head 2 just proposed. The resulting proposal factorizes across positions rather than forming an autoregressive chain. This weakens long-horizon draft quality and usually limits acceptance length. Medusa addresses this with a candidate tree. Each head emits top$-k$ candidates, and their Cartesian product defines possible continuations. A tree-structured attention mask lets the target verify many branches in one pass while each node attends only to its ancestors. Shared prefixes share compute, letting you explore many candidate continuations per cycle instead of betting on a single greedy draft. Medusa searches for a sparse fixed tree under a node budget. It favors combinations with higher estimated acceptance probability and prunes low-value nodes ^[2].

One caveat: exactness comes from rejection-sampling verification, not from the drafter. Medusa also defines a typical-acceptance heuristic based on a target-probability threshold. This heuristic does not preserve the exact target distribution. The paper reports higher acceptance with this heuristic. Use standard rejection sampling when the exact target distribution is required ^[2].

We’re now onto the important question of how the heads get trained, which is where Medusa-1 and Medusa-2 differ. Medusa-1 freezes the backbone (your target model) and trains only the heads on top. It’s cheap, stable and doesn’t affect the quality of the target because those weights are never touched. The downside is that the backbone’s features weren’t really trained to support these heads, so the later heads in particular stay weak and acceptance is left on the table. Medusa-2 unfreezes the backbone and trains it jointly with the heads, which improves the features the head depends on, pushing acceptance higher. The obvious risk here is that we can end up degrading the target model itself. The paper keeps the backbone from drifting by using three tricks:

1. a combined loss that keeps the original next-token objective alive alongside the head loss
2. a smaller learning rate on the backbone than on the heads,
3. and a warmup that trains the heads first before letting any gradients reach the backbone ^[2].

Both variants can train on data aligned with the target model. Self-distillation is an optional fallback when the original fine-tuning data is unavailable.

Medusa favors very low draft cost. It adds no separate draft-model pass. Parallel heads and tree verification keep the added computation small. Its independent heads have less conditional information than an autoregressive drafter, so later positions are harder to predict. And that is exactly what EAGLE aims to address.

EAGLE 1, 2 & 3

Medusa’s later heads cannot see the tokens proposed by earlier heads. The direct fix is to condition each draft step on the previous sampled token. A standalone autoregressive drafter already does this, but it pays for several sequential model passes per speculation cycle. EAGLE keeps the autoregressive chain but predicts in the target model’s feature space. EAGLE stands for Extrapolation Algorithm for Greater Language-model Efficiency.

EAGLE-1

The authors’ key observation is that next-step prediction is easier in the target model’s feature space than directly in token space. Predicting the next token from scratch is hard; predicting the next hidden state, given the states you’ve already seen, is relatively easier to extrapolate. The target has already computed a rich representation of the accepted prefix. EAGLE therefore drafts from the target’s second-to-top-layer features, which the paper defines as the hidden states immediately before the LM head. This lets the drafter reuse hidden states that the target has already computed ^[4].

The EAGLE drafter uses the target embedding and LM head, plus a trainable autoregression head with one projection and one Transformer decoder layer. It takes the feature sequence, extrapolates the next feature, and then borrows the target’s own LM head to turn that feature into a token distribution. A predicted feature is ambiguous unless the drafter also knows which token was sampled at the previous step. EAGLE therefore feeds the one-step-shifted sampled token sequence alongside the feature sequence. This identifies the sampled branch used for the next prediction. This reduces feature uncertainty ^[4]. Tree drafting changes which candidates are proposed. Exact speculative-sampling verification is what preserves the target distribution.

EAGLE-2

EAGLE-1 verifies a static draft tree with the same shape at every step. EAGLE-2 keeps the same weights and only changes the tree. It builds on the insight that a draft token’s acceptance probability isn’t fixed; it depends on context and on its position in the tree, and the draft model’s own confidence scores turn out to be a good cheap proxy for it. So instead of a fixed shape, EAGLE-2 will grow the tree dynamically, expanding branches on nodes the drafter is confident about, and pruning the ones it isn’t. During expansion, EAGLE-2 scores each frontier node by the product of draft probabilities along its path. It expands the highest-scoring nodes, then reranks all candidates and keeps a fixed verification-node budget. This is inference-time tree construction, so it needs no retraining. It keeps the same draft model and a fixed node budget, but adds tree-selection logic ^[?].

EAGLE-3

One limitation the authors identify is that EAGLE-1 and EAGLE-2 stop benefiting much from additional training data. EAGLE-3 traces this to the feature-prediction loss. EAGLE-1's loss has two parts. One trains the draft head to predict the next token. The other, the feature-prediction loss, trains it to match a specific hidden state of the target. This feature-prediction loss is an additional constraint. The task we care about is predicting the token, and matching one exact hidden state is stronger supervision than that task needs. In EAGLE-3 the paper's argument is that this constraint limits the draft model's expressiveness and is what caps its gains from more data ^[5].

EAGLE-3 therefore removes the feature-prediction loss and optimizes only the token-prediction objective. Removing the feature-prediction loss has two consequences. First, the feature-prediction loss no longer dictates what information the drafter should consume. Under the old loss the head had to reproduce the target's top-layer feature, so that feature was its natural input. The feature before the LM head is closely aligned with the next-token logits. It may not expose the best signal for tokens several steps ahead. With the constraint gone, EAGLE-3 fuses features from low, middle, and high layers of the target, giving the drafter a richer view of the target’s internal representations. Second, EAGLE-3 has to replace something the old loss used to provide. Because the draft was explicitly trained to reproduce the target’s hidden features, those predicted features remained meaningful inputs for the next drafting step. Removing the loss breaks this.

At inference, the first draft step reads fused target features. Later steps read the drafter’s own previous output, so the input distribution changes with depth. The paper’s training-time test procedure reduces this mismatch by feeding draft outputs back as later-step inputs during training. In the paper’s experiments, EAGLE-3 continued to improve with more training data and outperformed EAGLE-2 under the reported settings. Importantly, EAGLE-3 still drafts autoregressively over hidden representations during inference. The tradeoff is a more involved training procedure and a target-specific dependency on features from several target layers ^[5].

P-EAGLE

EAGLE-1 and EAGLE-3 improve the drafter, while EAGLE-2 improves tree construction. All three still generate draft states sequentially. To speculate $K$ tokens, the EAGLE head runs K times sequentially. On fast hardware with a large $K$, these $K$ sequential passes can start adding up. P-EAGLE, or Parallel EAGLE, aims to emit all $K$ draft tokens in one forward pass instead of K sequential draft passes. Architecturally, P-EAGLE resembles Medusa. One forward pass produces $K$ draft positions in parallel. The difference is what each position receives as input.

EAGLE’s input for each draft step is a (feature, shifted-token) pair, the target’s hidden state plus the token that it generated. For the first position, P-EAGLE does exactly the same - the real token the target emitted and hidden state behind it. For positions $2$ through $K$, the future tokens and hidden states do not yet exist. P-EAGLE uses learned placeholders for them. The placeholders are one shared learned token embedding and one shared learned feature vector. Positional information still distinguishes the future slots. The first slot uses the real target state. Later slots use the learned placeholders. One draft-model pass produces all $K$ positions. Removing direct access to the previous draft state makes the prediction task harder at fixed model capacity. A deeper parallel drafter can recover some or all of that gap. The tradeoff is a deeper single pass instead of $K$ smaller sequential passes. Whether this lowers end-to-end latency depends on $K$, model size, and hardware utilization ^[6].

The complex part of parallel drafting is making training scalable. A parallel multi-token head has higher memory requirements than a sequential one. To supervise $K$ parallel predictions, each training sequence of length $n$ has to be unrolled into roughly $n\timesK$ positions, one copy of the sequence per prediction depth and attention over them costs $O((nK)^2)$. Reasoning models routinely generate thousands of tokens, so the drafter must be trained on long contexts. Unfortunately, this is exactly where the quadratic memory cost becomes prohibitive. P-EAGLE makes long-context training feasible with two tricks. First, it precomputes the attention mask. The cross-depth causal structure is position-invariant. P-EAGLE therefore builds one maximum-length mask and slices the required top-left submatrix for each example. The second is sequence partitioning. When one sequence still does not fit, P-EAGLE partitions it, runs one forward-backward pass per segment, and accumulates gradients across segments. This is gradient accumulation within one sequence.The partitioning preserves each retained position’s required dependency chain. With $S$ segments, peak attention memory drops from $$O(L^2) to $O(\frac{L^2}{S^2})$ ^[6].

P-EAGLE trades a deeper single draft pass for fewer sequential draft passes. Its goal is to reduce draft latency while retaining the acceptance gains of a larger drafter. In the paper’s four-layer configuration, P-EAGLE matched or exceeded one-layer autoregressive EAGLE-3 in acceptance length and improved throughput by $1.10\times$ to $1.36\times$. The gain depends on model size, speculation depth, and concurrency. At small $K$, the deeper parallel pass can be slower ^[6].

PARD 1 & 2

P-EAGLE keeps EAGLE’s target-conditioned, feature-fed head intact, but replaces K sequential draft passes with one parallel pass. PARD, by contrast, changes the drafter itself. Rather than another component fed by the target, PARD's drafter is a standalone small language model that does not consume target-model features. This target independence lets one trained drafter serve several compatible targets in the same model family.

PARD 1

PARD starts from an existing, capable small model (e.g. Llama 3.2-1B for a LLaMA 3.1-8B target) and adapts it to draft in parallel. The drafter is already a competent LM independent of any one target, which lets it be reused across compatible models in the family. Like P-EAGLE, PARD produces K draft tokens in a single forward pass. The drafter appends a run of special mask tokens (placeholders) to the prefix and predicts each future position conditioned on the real prefix and preceding mask placeholders. All K tokens can thus be computed in parallel because no drafted position depends on another drafted token. Target independence comes from conditioning on token inputs rather than target-model features.

PARD adapts the autoregressive model, which has no notion of a mask placeholder, through fine-tuning. It splits the objective into K subtasks, one per future offset, and trains them together over the same sequence. The offset-k subtask learns to predict the true token k positions ahead from the prefix and the k-1 placeholders that stand in for the intervening tokens, using a cross-entropy loss. A purpose-built attention mask lets all K subtasks share a single pass over the sequence and keeps the conditioning pattern consistent between training and inference. The fine-tuning teaches the autoregressive drafter to treat mask positions as parallel prediction slots ^[7].

Splitting a length-N sequence into K prediction offsets expands N training positions to roughly $K\timesN$, so cost scales with the number of prediction offsets. The authors use a COnditional Drop-token (COD) mechanism to bring the training cost down. COD keeps a shrinking fraction of positions at each successive offset. COD retains a training position only when all key and value states required by that position remain present. With COD, the retained-token count follows a geometric schedule instead of scaling directly as$K\timesN$ ^[7].

PARD trades target-specific feature conditioning for reuse across compatible targets. Its standalone drafter is larger than the lightweight heads used by EAGLE-style methods, but it does not require target features. Parallel prediction removes $K$ sequential draft passes. End-to-end gains still depend on drafter size, acceptance length, batch size, and serving implementation.

PARD 2

PARD-1 is reusable across compatible targets, but it cannot exploit target features when they are available. With PARD-2, target conditioning becomes optional at inference. It is a dual-mode drafter that can run target-independently as before or target-dependently when given fused target hidden features as additional input. During training, stochastic gating sometimes removes these features so that the drafter learns to operate in both modes. It also changes how the token-level training loss is weighted. The authors argue that token-prediction accuracy alone does not match the inference-time objective. The relevant quantity is how many consecutive draft tokens survive verification.

They introduce Confidence-Adaptive Token (CAT) optimization, which weights each token by the cumulative target confidence of its preceding prefix, used as a proxy for the probability that verification reaches that position. The objective still uses token-level losses, but weights them by their expected contribution to acceptance length. PARD-2 applies these weights to both the cross-entropy and knowledge-distillation losses. Unlike the fixed, position-based weights used in related parallel-draft work, the reweighting is adaptive and depends on the prefix rather than position alone ^[8]. The target-dependent mode requires model-specific feature extraction and training. The target-independent mode is more reusable, but it does not use target features at inference.

DFlash

Parallel drafting lowers latency, but draft quality can still limit acceptance length. Like PARD, DFlash drafts an entire block in one forward pass. Unlike PARD, it uses a small block-diffusion model conditioned on features from the target. The model fills all masked positions in parallel. Because all positions are decoded in parallel, drafting latency changes little for moderate block sizes. In the authors’ latency test, a five-layer DFlash drafter generated a 16-token block faster than a one-layer EAGLE-3 drafter generated 8 tokens. It also had a longer average acceptance length in that setup ^[9].

DFlash extracts hidden features from a fixed set of target layers spread from shallow to deep. A lightweight projection fuses them into one compact context feature. The key difference from EAGLE is where this feature enters the drafter. EAGLE mixes the target features with the input token embeddings. The DFlash authors argue that this signal weakens with depth, so extra layers help less. DFlash instead injects the fused feature into the key and value projections of every draft layer. The projected features are stored in the draft KV cache and reused across drafting iterations. This keeps the conditioning signal available at every layer. In the paper’s ablation, acceptance length increased with draft depth, while drafting latency also increased.

DFlash adapts block-diffusion training to the speculative setting. Instead of using fixed blocks, DFlash samples anchor tokens from the response. Each anchor becomes the clean first token of a block. The remaining positions are masked and predicted in parallel. This matches inference. Each block starts from the clean bonus token produced by the target during the previous verification step. The loss gives more weight to earlier positions. An early error prevents later tokens from being accepted. The drafter shares the target’s token embedding and LM head. Both remain frozen. Only the draft transformer layers are trained. This keeps the drafter lightweight and aligned with the target’s representation space ^[9].

DFlash uses one-pass drafting to lower latency and target conditioning to improve acceptance length. The tradeoff is tight coupling to the target. The drafter depends on target features at every layer and shares the target’s embedding and LM head. It is therefore less reusable than PARD’s target-independent mode.

Multi-Token Prediction (MTP)

Every method so far adds a draft path after the target model has been trained. Multi-token prediction makes the draft path part of the training design. The added modules learn to predict future tokens and can later be reused for drafting. The drafter comes out of how the model was trained.

DeepSeek-V3 MTP

DeepSeek-V3 adds MTP as an extra objective during pretraining. Alongside the usual next-token objective, the model predicts additional future tokens. The extra losses are averaged across prediction depths, scaled, and added to the main loss. The authors give two reasons. First, each position now supervises more than one prediction. Second, it may encourage the model to prepare its representations for later tokens ^[10].

Each prediction depth gets its own module. Each module reuses the main model’s embedding table and output head. It adds one Transformer block and one projection. Predicting D additional tokens requires D modules. Each module has its own block and projection. DeepSeek-V3 uses a depth of one. During training, its single module combines the main model’s hidden state with the ground-truth embedding of the next token. It then predicts the token after it. For larger depths, the modules run in sequence. Each module reads the representation from the previous depth and the token embedding for its own offset. This preserves the causal chain across depths. Each module has separate weights for its prediction depth. By contrast, Medusa reads all predictions from one shared hidden state. Its heads cannot condition on one another. DeepSeek’s sequential modules let each depth build on the previous one ^[10].

The main goal is to improve the target model. During normal inference, the MTP module can be removed and the target can run by itself. The same module can also be kept for speculative decoding. DeepSeek-V3 uses one MTP depth, so the retained module supplies one additional future-token prediction for speculative verification. The drafting path reuses what was trained to improve the target. The tradeoff is extra parameters and training cost, even when the module is removed at inference.

Gemma4 MTP drafter

Unlike DeepSeek-V3, Gemma 4’s MTP module is designed for speculative decoding. It ships as a separate small drafter for each matching target. The drafter is a four-layer Transformer head with three local-attention layers and one global-attention layer. The decoder itself is simple. The main optimizations reuse target state and reduce the cost of the output layer. On the first draft step, the head combines the target’s previous-step last-layer activation with the embedding of the current token, then projects the result to the drafter width. Later steps combine the previous MTP activation with the newly sampled token embedding. This creates an autoregressive draft chain. The target activation seeds the first draft step. Each MTP layer also cross-attends to the target model’s KV states. The report specifies a separate MTP embedder ^[11].

The MTP head cross-attends to the target model’s KV states instead of running a separate prefill over the prompt. Instead, its attention layers read the target’s existing KV cache. The local draft layers use the target’s last local-attention cache. The global draft layer uses the target’s last global-attention cache. This removes a separate MTP prefill and supports arbitrary draft length. It is the main difference from a standalone autoregressive drafter.

The final step maps the drafter’s hidden state to scores over the vocabulary. Gemma 4 has a 262,144-token vocabulary, so this requires a large matrix multiplication. For the small drafter, this output projection becomes a large part of the total cost. The E2B and E4B drafters avoid scoring the full vocabulary. The vocabulary is partitioned into clusters. At runtime, the drafter first scores the clusters. It then scores only the tokens inside the most likely clusters. Google reports a similar acceptance rate while reducing the cost of the output projection. This lowers output-head cost for the E2B and E4B drafters, but the drafter remains target-specific. It depends on target activations and target KV states ^[11].

DSpark

DFlash lowers draft latency, but its parallel positions cannot condition on earlier sampled tokens in the same block. DSpark’s key insight is that a small sequential correction can restore local token dependencies without making the full drafter autoregressive. It adds a lightweight sequential head after the parallel backbone and a hardware-aware scheduler before verification. The tradeoff is a short serial sampling loop, and the system still pays to draft the full block ^[13].

The authors use a simple example. A block might continue as “of course” or “no problem.” Independent positions can mix them into “of problem” or “no course.” This causes acceptance to drop at later positions. The authors call this suffix decay. DSpark uses semi-autoregressive drafting. The parallel backbone runs once and produces base logits for every position. The sequential head adds a transition bias before each token is sampled. Sampling proceeds from left to right. With the default Markov head, position k depends on the token sampled at position k-1. The head is small, so the parallel pass still dominates draft latency. The default is a first-order Markov head. It approximates the full transition matrix with a rank-256 factorization. This keeps storage and per-step compute small. An RNN variant instead carries state across the full block prefix.

The second mechanism concerns verification, not drafting. For each position, a confidence head predicts the probability that the token will be accepted, assuming all earlier tokens were accepted. Its training target is one minus the total variation distance between the draft and target distributions. The raw confidence scores are often too high. Sequential temperature scaling calibrates the cumulative prefix probabilities against observed acceptance rates. It preserves the ranking of the confidence scores. The scheduler combines these calibrated probabilities with a throughput curve measured for the serving engine. It then chooses one verification length per request to maximize expected token throughput across the batch. To preserve exact sampling, the greedy search stops when adding the next candidate lowers estimated throughput. This prevents later draft tokens from influencing an earlier truncation decision. DSpark still drafts the full block. The scheduler removes the unused suffix before target verification. DSpark shares the target’s embedding and LM head, and keeps both frozen. Training updates only the parallel backbone, sequential head, and confidence head. The loss combines token prediction, target-distribution matching, and confidence prediction. Each term gives more weight to earlier block positions ^[13].

In the paper’s offline Qwen3 experiments, DSpark reports a higher average accepted length than its EAGLE-3 and DFlash baselines across math, code, and chat. In its DeepSeek-V4 deployment, the paper reports 60 to 85 percent higher per-user generation speed for V4-Flash and 57 to 78 percent for V4-Pro than the MTP-1 baseline at matched aggregate throughput. The production result is not one fixed speedup. The benefit changes with system load. As load rises, the scheduler shortens verification to avoid spending batch capacity on unlikely suffix tokens. The sequential head improves draft coherence, and the scheduler reduces wasted target verification. DSpark still pays the fixed cost of drafting the full block and the serial cost of the small head ^[13].

References