token-constraints
Structured output with speculative decoding
A grammar matcher masks every token that would break the required format, and speculative rounds run it on a private copy that follows each draft and rewinds when drafts are rejected.
The problem
- Structured output, such as a reply that must match a JSON schema, forbids most tokens at each position, and which ones depends on everything written so far.
- A speculative round drafts several tokens before any of them is accepted, so the position after the first draft needs the grammar state that draft would leave.
- A rejected draft must not leave the grammar state ahead of the text, or every later mask would be wrong.
How it works
- A matcher tracks where the committed output stands in the grammar. Its allowed set masks every other token before temperature, top-k, top-p and min-p see the row.
- A speculative round forks the matcher together with the model state, so drafts move a private copy and never the committed one.
- Each draft position is sampled under the mask of the fork's state after the drafts before it, and the verify pass masks each position the same way.
- Acceptance and the correction use both models' masked laws, so a correction drawn from max(p − q, 0) stays inside the language.
- On a rejection the fork rewinds to the accepted drafts and takes the correction, and only then becomes the committed state.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
rounds | native | native, torch | what samples inside captured decode rounds (rounds:sampler): token-sampling's native round form (its code/sampling.cu built for the device at load; where it does not build, the torch law, with a warning) or its torch law |
configuration.json
{"plugins": {
"use": {"sampler": "token-constraints"},
"options": {"token-constraints": {"rounds": "native"}}
}}Pays off when
- Requests ask for JSON, a JSON schema, a regular expression or a fixed set of choices, and the engine speculates with sampled drafts.
- Constrained requests should keep drafting instead of falling back to one token per pass.
Doesn't pay off when
- Decoding is one token at a time; masking with the matcher's allowed set is all it needs, without forks.
- Output is unconstrained, so there is nothing to mask.
- Every position reads the matcher's allowed set and every append copies the matcher's history, work that unconstrained drafting does not do.
What must stay true
- The mask applies before the probability filters, and acceptance and correction use each model's actual masked law.
- The matcher and the model always describe the same accepted output; a rejected or failed branch never publishes its matcher state.
- The end token is allowed only once the output is complete, and nothing may be appended after it.
crit-hub