crit-hub

chain-verifier

Chain verification

A round keeps the longest prefix of the draft chain the model agrees with, then the model's own token: the correction where a draft failed, a bonus at the end.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Speculative decoding drafts a few tokens, then one pass of the model predicts the token after each of them. Something has to decide which drafts stand.
  • Keeping a draft the model would not have written changes the text, and keeping too few wastes the round.

How it works

  1. The model's pass gives its own prediction after the pending token and after each draft.
  2. The first draft stands if it equals the model's prediction at its place, the next if it equals the prediction after that, and so on.
  3. At the first draft that differs, the model's own token takes its place (the correction). If every draft stood, the model's token after the last one is added (the bonus).
  4. The round never emits more than the request's output allowance, and it ends right after a stop token.
  5. It verifies chains only, so a tree of drafts is refused (it conflicts with draft:tree).

Configuration

No options: an engine selects it, and that is the whole setting.

configuration.json
{"plugins": {
  "use": {"verifier": "chain-verifier"}
}}

Pays off when

  • The drafter proposes one chain a round, as greedy-head-drafter and mtp-decode do.
  • A tree verifier is checked: with one branch, it must keep what this one keeps.

Doesn't pay off when

  • The drafter proposes a tree of drafts. draft-trees fills the verifier slot too, with a verifier of its own.

What must stay true

  • A draft stands only if every draft before it stood and it equals the model's prediction at its place.
  • Every round gives at least one token: the correction or the bonus.
  • No token past the output allowance or after a stop token is emitted.