crit-hub

greedy-head-drafter

MTP speculative decoding

The model's own multi-token prediction layer guesses the next few tokens, and one pass of the full model checks them all at once.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Decoding writes one token per pass, and every pass reads all the weights, so the GPU mostly waits on memory.
  • A pass that checks five positions costs about the same as one that checks one.

How it works

  1. The MTP layer, a small extra layer the checkpoint ships with, drafts a few tokens one after another. Each draft costs a fraction of a full pass.
  2. The full model runs once over the last token and every draft, and predicts the token after each of them.
  3. Drafts are kept up to the first one the full model disagrees with.
  4. The full model's own prediction at that point is added for free, so every round gives at least one token.
  5. With sampling, rejection sampling decides which drafts stand, so the output follows exactly the distribution plain decoding would give.

Configuration

No options: an engine selects it, and that is the whole setting.

configuration.json
{"plugins": {
  "use": {"drafter": "greedy-head-drafter"}
}}

Pays off when

  • The checkpoint ships an MTP layer, and drafts are often right (code, tool calls, long agent turns).
  • One request at a time, where a pass is bound by reading the weights.

Doesn't pay off when

  • Drafts are rarely right, so the extra passes and the repair cost more than they save.
  • Experts stream from host memory, because checking several tokens loads the union of their experts.

What must stay true

  • Only accepted tokens reach the output, the KV cache and the recurrent state; a rejected draft leaves no trace.
  • Sampled output follows the same distribution as plain decoding. Greedy output equals plain greedy decoding token for token.
  • The drafter is used as the checkpoint publishes it, never trained on text like the benchmark's.