crit-hub

draft-vocabulary-subset

Draft vocabulary subset

The drafter scores only the output-head rows of the tokens the model writes most, so drafts get cheaper, while verification keeps the full head and greedy output is unchanged.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Every draft step multiplies the drafter's hidden state with the whole output head, one row per token, and vocabularies run past 100,000 tokens.
  • For a small drafter such as an MTP layer, that head is a large share of what a draft costs.

How it works

  1. Count the tokens the full model writes on a broad, mixed set of prompts, and keep the rows of the most frequent ones plus required tokens such as the end token.
  2. Each draft step scores only those rows, read in place from the existing head, and maps the winning row's subset index back to its token ID.
  3. One pass of the full model checks the drafts with the full head, as in any speculative round, and drafts stand up to the first one it disagrees with.
  4. A token outside the subset can never be drafted, so the draft there misses and the model's own token takes its place. The output does not change; only how many drafts stand does.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
size1024int 1 to 1048576Rows of the draft head scored
refresh64int 1 to 1000000rounds between choosing the subset again from emitted counts
configuration.json
{"plugins": {
  "use":     {"drafter": "draft-vocabulary-subset"},
  "options": {"draft-vocabulary-subset": {"size": 1024, "refresh": 64}}
}}

Pays off when

  • The drafter is small, so its output head is a large share of each draft step.
  • A modest subset covers most of the tokens the workload produces.

Doesn't pay off when

  • Drafting is dominated by other work, or the head is already cheap next to verification.
  • The text often needs tokens outside the subset, such as another language. In one engine, a subset built from mostly English text cut tokens per round in Chinese, Japanese, Russian and German.
  • Tokens spread evenly across the vocabulary, so no small subset covers them.

What must stay true

  • Verification uses the full head, and only the full model accepts tokens.
  • Subset rows are the head's own rows, read byte for byte with their scales and never re-encoded.
  • Rows are unique and ascending, so subset index order is token ID order. Drafter and model share token IDs and break ties the same way, by taking the first maximum.
  • The subset is chosen from broad, multilingual text and its size from measured rounds, never from a benchmark score.