crit-hub

sequential-recurrence

Gated-delta recurrence

The gated-delta recurrence token by token, keeping every state, with long prompt rows through a chunked kernel.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A gated-delta layer, the recurrent layer of hybrid models, folds each token into one state matrix per head instead of keeping a row per token.
  • The state depends on every token before it, in order, so every faster form (chunked kernels, recovery after a speculative round) must give the state this one gives.

How it works

  1. For each token, the state decays by the token's gate.
  2. The state predicts the token's value from its key. The difference from the real value, weighted by the token's beta, is written into the state (the delta rule).
  3. The token's output is the updated state read with its query.
  4. On the host it runs in FP64, one token after another. On the device it keeps every sequence's state in a pool.
  5. On the device, rows of up to exact_rows tokens (decode tokens, verify windows) run token by token and keep every state, and longer prompt chunks go through fla's chunked kernel when fla is installed.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
exact_rows512int 0 to 4096prompt rows of at most this many tokens run the exact column scan (one launch a layer, the bits of decoding the tokens one by one), longer ones the chunked kernel (fla's where installed, another summation order), in an eager call and in a captured prompt bucket alike; 0: every prompt row the chunked kernel. The two give other bits, so the configuration states the value and nothing measures it at load. The default, 512, is the largest of crit_plugins.prompt_graphs' default buckets: a captured bucket has the bits of the eager chunk of the same tokens, and a prompt cut into chunks of 1,024 tokens or more runs its last chunk as the full ones when that chunk is longer than 512 tokens. A model whose prompt chunks are at most this long runs every prompt row through the scan (a prefix's rows equal the whole prompt's). device_states.scan_rows() measures where the chunked kernel gets faster on a shape and device, to recommend a value
chunk_fp32truetrue, falselong rows reach fla's chunked kernel in FP32: closer to the exact recurrence where strong decays leave outputs small, at twice the kernel's time
commitdeferreddeferred, checkpoints, recomputehow a captured round keeps its accepted prefix: deferred (record the count; the next row replays it from a stash), checkpoints (the state after every column, one selected) or recompute (the start state kept, the prefix replayed at commit)
kernelstruetrue, falserun decode and verify rows through recurrent-scans from the kernel registry where it is installed; false: the torch formula
configuration.json
{"plugins": {
  "use":     {"recurrence": "sequential-recurrence"},
  "options": {"sequential-recurrence": {"exact_rows": 512, "chunk_fp32": true, "commit": "deferred"}}
}}

Pays off when

  • The model has gated-delta layers, as hybrid models such as Qwen3-Next do.
  • A recurrence plugin is checked: recurrent-state-recovery is compared with it.

Doesn't pay off when

  • Long prompts with fla not installed, which then run token by token.
  • chunk_fp32 is on and decays are weak, so the kernel's FP32 takes twice its time for little gain.

What must stay true

  • Every token updates the state once, in order, with one formula.
  • Short rows keep every state they pass through, so a speculative round can keep the state after exactly its accepted tokens.