draft-vocabulary-subset
Draft vocabulary subset
The drafter scores only the output-head rows of the tokens the model writes most, so drafts get cheaper, while verification keeps the full head and greedy output is unchanged.
The problem
- Every draft step multiplies the drafter's hidden state with the whole output head, one row per token, and vocabularies run past 100,000 tokens.
- For a small drafter such as an MTP layer, that head is a large share of what a draft costs.
How it works
- Count the tokens the full model writes on a broad, mixed set of prompts, and keep the rows of the most frequent ones plus required tokens such as the end token.
- Each draft step scores only those rows, read in place from the existing head, and maps the winning row's subset index back to its token ID.
- One pass of the full model checks the drafts with the full head, as in any speculative round, and drafts stand up to the first one it disagrees with.
- A token outside the subset can never be drafted, so the draft there misses and the model's own token takes its place. The output does not change; only how many drafts stand does.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
size | 1024 | int 1 to 1048576 | Rows of the draft head scored |
refresh | 64 | int 1 to 1000000 | rounds between choosing the subset again from emitted counts |
configuration.json
{"plugins": {
"use": {"drafter": "draft-vocabulary-subset"},
"options": {"draft-vocabulary-subset": {"size": 1024, "refresh": 64}}
}}Pays off when
- The drafter is small, so its output head is a large share of each draft step.
- A modest subset covers most of the tokens the workload produces.
Doesn't pay off when
- Drafting is dominated by other work, or the head is already cheap next to verification.
- The text often needs tokens outside the subset, such as another language. In one engine, a subset built from mostly English text cut tokens per round in Chinese, Japanese, Russian and German.
- Tokens spread evenly across the vocabulary, so no small subset covers them.
What must stay true
- Verification uses the full head, and only the full model accepts tokens.
- Subset rows are the head's own rows, read byte for byte with their scales and never re-encoded.
- Rows are unique and ascending, so subset index order is token ID order. Drafter and model share token IDs and break ties the same way, by taking the first maximum.
- The subset is chosen from broad, multilingual text and its size from measured rounds, never from a benchmark score.
crit-hub