crit-hub

Speculative rounds on the GPU

Speculation The GPU drafts, verifies, accepts and moves on in two captured graphs, so the host waits once per speculative round instead of deciding between steps.

In an engine

No slot holds it: a model plugin runs it as its native rounds, with the drafter, verifier and sampler plugins inside.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A host-driven speculative loop reads tokens back, compares drafts with predictions and updates its bookkeeping on the CPU, and the GPU idles through every round trip.
  • Launching each draft step, the verify and the commit separately adds launch overhead to every round.

How it works

  1. The GPU keeps the request's control block, with the frontier (how many positions are in the cache), the pending token and the round's emitted count. The host only keeps a mirror of it.
  2. The host launches the Forward graph, a captured CUDA graph that copies in the round's allowance (how many tokens it may emit), drafts a few tokens and verifies them in one pass.
  3. The host launches the Finish graph. It accepts on the GPU with the same rule a host loop uses, moves the frontier on by the emitted count and keeps the model state of the last accepted column.
  4. One readback brings a small packed block to the host, with the count, the tokens and the new frontier, which must equal the mirror plus the count.

Pays off when

  • A synchronous speculative loop is already correct, and host round trips, readbacks or small launches dominate the round.
  • Greedy and sampled requests alike, since rejection sampling writes predictions that the same accept step reads.

Doesn't pay off when

  • A verify pass over the drafts costs about as much as decoding each of those tokens one by one, so no round can pay.
  • The engine cannot keep the per-column state (KV columns, recurrent checkpoints or saved inputs) needed to select any accepted prefix.
  • Experts stream from host memory, where every verify column adds its own expert misses.

What must stay true

  • Acceptance on the GPU follows the host loop's rule exactly. Drafts stand up to the first mismatch, then the correction or bonus, within the allowance and the stop tokens.
  • Exactly one readback per round, and no host wait inside the launch.
  • The frontier moves on by the emitted count n, and the state kept is that of verify column n - 1, the last accepted input.
  • A smaller count from crit Core, for example at a stop string, is applied on the GPU before the next round, and selecting the state again gives the same result.