greedy-head-drafter
MTP speculative decoding
The model's own multi-token prediction layer guesses the next few tokens, and one pass of the full model checks them all at once.
The problem
- Decoding writes one token per pass, and every pass reads all the weights, so the GPU mostly waits on memory.
- A pass that checks five positions costs about the same as one that checks one.
How it works
- The MTP layer, a small extra layer the checkpoint ships with, drafts a few tokens one after another. Each draft costs a fraction of a full pass.
- The full model runs once over the last token and every draft, and predicts the token after each of them.
- Drafts are kept up to the first one the full model disagrees with.
- The full model's own prediction at that point is added for free, so every round gives at least one token.
- With sampling, rejection sampling decides which drafts stand, so the output follows exactly the distribution plain decoding would give.
Configuration
No options: an engine selects it, and that is the whole setting.
configuration.json
{"plugins": {
"use": {"drafter": "greedy-head-drafter"}
}}Pays off when
- The checkpoint ships an MTP layer, and drafts are often right (code, tool calls, long agent turns).
- One request at a time, where a pass is bound by reading the weights.
Doesn't pay off when
- Drafts are rarely right, so the extra passes and the repair cost more than they save.
- Experts stream from host memory, because checking several tokens loads the union of their experts.
What must stay true
- Only accepted tokens reach the output, the KV cache and the recurrent state; a rejected draft leaves no trace.
- Sampled output follows the same distribution as plain decoding. Greedy output equals plain greedy decoding token for token.
- The drafter is used as the checkpoint publishes it, never trained on text like the benchmark's.
crit-hub