recurrent-state-recovery
Recurrent state recovery
A recurrent layer mixes every token into one state, so after a speculative round the engine rebuilds that state for exactly the accepted tokens instead of dropping rejected rows.
The problem
- A gated delta layer, a recurrent layer used in hybrid models, folds each token into one fixed-size state matrix per head. There are no per-token rows to drop.
- Verify runs the pending token and every draft through that state, so afterwards it also holds the drafts that were rejected.
- Saving the whole matrix after every draft multiplies the state's memory by the draft width.
How it works
- Verify runs the state forward through every column in a running copy. The live state stays as it was when the round began.
- Attention just drops the rejected KV rows. The recurrent layer needs the state after exactly the accepted tokens, so it rebuilds it from what verify kept.
- Checkpoints: verify saves the state after every column, and commit copies the accepted one into the live state.
- Recompute: verify keeps the round's starting state and each token's small inputs, and commit replays the accepted steps from that start.
- Deferred: verify keeps only a small stash per column, commit writes just the accepted count, and the next verify replays those steps first.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
chunk_size | 64 | int, at least 1 | the host form's chunk |
exact_rows | 512 | int 0 to 4096 | prompt rows of at most this many tokens run the exact column scan (one launch a layer, the bits of decoding the tokens one by one), longer ones the two-pass prefill (another summation order), in an eager call and in a captured prompt bucket alike; 0: every prompt row the two-pass prefill. The two give other bits, so the configuration states the value and nothing measures it at load. The default, 512, is the largest of crit_plugins.prompt_graphs' default buckets (see sequential-recurrence's exact_rows); device_states.scan_rows() measures where the two-pass prefill gets faster on a shape and device, to recommend a value |
commit | deferred | deferred, checkpoints, recompute | how a captured round keeps its accepted prefix: deferred (record the count; the next row replays it from a stash), checkpoints (the state after every column, one selected) or recompute (the start state kept, the prefix replayed at commit) |
kernels | true | true, false | run rows through recurrent-scans from the kernel registry where it is installed; false: the torch formula |
configuration.json
{"plugins": {
"use": {"recurrence": "recurrent-state-recovery"},
"options": {"recurrent-state-recovery": {"chunk_size": 64, "exact_rows": 512, "commit": "deferred"}}
}}Pays off when
- Hybrid models with gated delta layers that decode speculatively, where the state is large and the accepted prefix short.
- When memory is tight, recompute and deferred keep at most one extra state copy instead of one per column.
Doesn't pay off when
- Replaying the accepted steps costs more than storing a state per column would.
- The rest of the continuation state, such as convolution windows, cannot be restored along with it.
What must stay true
- Decode, verify, prefill and replay compute a step with one update expression in the same order, so every way gives the same bits.
- The state at boundary B is the state after B materialized tokens. A correction token counts only once it has been fed in.
- Commit is all or nothing. On a failure or a cancellation the committed state stays as it was.
- On the GPU the accepted count is read from device memory, and commit can run again with a smaller count after a truncation.
crit-hub