hidden-state-drafters
Hidden-state drafters
A small published drafter reads hidden states from a few of the target model's layers and guesses the next tokens, one per step or a whole block per pass.
The problem
- Most models ship no draft layer of their own, and copying continuations from earlier text rarely gets several tokens right.
- A drafter that guesses one token per step needs several small passes per round, and with one request each pass is bound by launch time more than by memory.
How it works
- The target keeps its hidden state after a few chosen layers, the taps, at every position it reads. Tap k is the state after k layers.
- EAGLE-3's drafter is one layer. It reads the taps at a position with the token at the next position and guesses the token after that. Later steps in the round feed it its own hidden state instead.
- One target pass verifies the drafts as in MTP decoding and writes fresh taps for every column.
- Before drafting again, the drafter catches up on the taps of exactly the accepted columns. It always runs the same number of columns, and the GPU's count masks the rest.
- A block drafter (DFlash, DSpark) instead puts the pending token and a few mask slots in one block, and fills every slot in a single pass with the target's taps as context.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
cut_target | 0 | float 0 to 0.99 | acceptance a kept draft must reach; 0 keeps every draft |
configuration.json
{"plugins": {
"use": {"drafter": "hidden-state-drafters"},
"options": {"hidden-state-drafters": {"cut_target": 0}}
}}Pays off when
- A drafter trained for the target is published (EAGLE-3, DFlash or DSpark checkpoints), and the target has no draft layer or its own accepts less.
- One request at a time, where a block drafter's single pass replaces several launch-bound draft steps.
Doesn't pay off when
- No drafter is published for the target. crit uses drafters as released and never trains one.
- The extra tap writes in every verify and the drafter's own passes cost more than the tokens they add.
What must stay true
- Taps are counted the way crit Core's drafter reader gives them, as the hidden state after k target layers, never re-derived from a format's own numbering.
- The drafter pairs the target's hidden state at a position with the token at the next position.
- Catch-up takes in exactly the accepted columns, using the count the GPU settled after any truncation.
- Drafts from a smaller draft vocabulary are mapped to target token ids before verification.
crit-hub