chain-verifier
Chain verification
A round keeps the longest prefix of the draft chain the model agrees with, then the model's own token: the correction where a draft failed, a bonus at the end.
The problem
- Speculative decoding drafts a few tokens, then one pass of the model predicts the token after each of them. Something has to decide which drafts stand.
- Keeping a draft the model would not have written changes the text, and keeping too few wastes the round.
How it works
- The model's pass gives its own prediction after the pending token and after each draft.
- The first draft stands if it equals the model's prediction at its place, the next if it equals the prediction after that, and so on.
- At the first draft that differs, the model's own token takes its place (the correction). If every draft stood, the model's token after the last one is added (the bonus).
- The round never emits more than the request's output allowance, and it ends right after a stop token.
- It verifies chains only, so a tree of drafts is refused (it conflicts with
draft:tree).
Configuration
No options: an engine selects it, and that is the whole setting.
configuration.json
{"plugins": {
"use": {"verifier": "chain-verifier"}
}}Pays off when
- The drafter proposes one chain a round, as
greedy-head-drafterandmtp-decodedo. - A tree verifier is checked: with one branch, it must keep what this one keeps.
Doesn't pay off when
- The drafter proposes a tree of drafts.
draft-treesfills the verifier slot too, with a verifier of its own.
What must stay true
- A draft stands only if every draft before it stood and it equals the model's prediction at its place.
- Every round gives at least one token: the correction or the bonus.
- No token past the output allowance or after a stop token is emitted.
crit-hub