mtp-decode
Prompt lookup decoding
Drafts are copied from wherever the last few tokens appeared earlier in the prompt or output, and one pass of the full model checks them all.
The problem
- Decoding writes one token per pass of the full model, and every pass reads all the weights.
- Edits, rewrites and answers about a pasted text repeat long runs of the prompt word for word, so the next tokens are often already in the context.
How it works
- The engine indexes every short run of tokens (two here) in the prompt and in what the model has written so far.
- Each round it looks up the last two tokens and copies the tokens that followed an earlier occurrence as drafts, up to four. A match in the model's own output must agree on a longer run, and the most recent match is preferred.
- One pass of the full model checks every draft and predicts the token after each. Drafts stand up to the first one it disagrees with, and its own token there is added, or a bonus token when all of them match.
- When the last two tokens appear nowhere earlier, there is nothing to copy and the round is an ordinary decoding step.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
window | 3 | int 1 to 64 | Tokens at the context's end matched to find a copy |
history_window | 6 | int 1 to 256 | Tokens of earlier output searched for a continuation |
capacity | 16384 | int 66 to 262144 | tokens of each round slot's history ring on the device: copies search its last capacity - (widest + 1) (the host index holds the whole context); more than the history window and 64 past the widest round |
override | 0 | int 0 to 64 | 0: copy drafts alone; N >= 1: the model's draft head drafts the chain and a copy whose match covers at least N tokens replaces its first drafts |
configuration.json
{"plugins": {
"use": {"drafter": "mtp-decode"},
"options": {"mtp-decode": {"window": 3, "history_window": 6, "capacity": 16384}}
}}Pays off when
- The output copies much of its input, as when editing code or documents, rewriting text or quoting a source.
- The checkpoint has no drafter. Copying needs no extra weights and no extra model pass.
- One request at a time, where a pass that checks five positions costs about what a pass for one does.
Doesn't pay off when
- The output is new text. Lookups rarely match, and a wrong copy still costs its verification and a rollback.
- Experts stream from host memory, because checking several tokens loads the union of their experts.
What must stay true
- Only the full model accepts tokens. Each checked position gets exactly the logits a one-token step would, so greedy output equals plain greedy decoding token for token.
- Only accepted tokens and the model's own corrections reach the output, the model's state and the index; a rejected draft leaves no trace.
- With sampling, drafts are accepted by probability (rejection sampling), not by matching, so the output still follows the model's distribution.
- Window lengths are tuned on general text, never on the benchmark.
crit-hub