crit-hub

Prefix checkpoints across tiers

Memory and loading A prompt prefix's computed state is saved as one complete bundle on the GPU, in host memory or on disk, and restored instead of recomputed when that is faster.

In an engine

No slot holds it: a model plugin calls it to save and restore prefix state.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Long prompts often start the same way, but the saved state of every such prefix is too large to keep on the GPU.
  • State kept in host memory or on disk only helps if it is found by its exact identity and brought back in time, without slowing the requests already running.
  • A bundle that is half copied, misses a component or is freed while it is read gives the model wrong state without any error.

How it works

  1. A checkpoint is keyed by its exact prefix tokens, a fingerprint of the weights and state layout, and its codec, the format the state is written in. It is one bundle with every piece of state needed to continue: here KV and recurrent state.
  2. For a new prompt, only bundles whose tokens are an exact prefix and whose fingerprint and codec match are candidates. The plan compares copy and restore time plus the remaining prefill with recomputing.
  3. A bundle in a slower tier is copied up while the request waits, a few windows of bytes between compute steps, but only if it lands within the request's time limit and still wins.
  4. A copied bundle is published only when every component has landed and its digests match. Restoring copies it into new, private GPU state, all components or none.
  5. Every reader holds a lease, a claim that keeps the bundle's bytes in place. Evicting a bundle hides it from new lookups, but its bytes stay until the last lease is closed.

Pays off when

  • Requests repeat long prefixes (system prompts, documents, chat history), and restoring their state is faster than prefilling it.
  • A bundle in a slower tier can be copied up within the request's time limit and still beat starting now from a shorter prefix.

Doesn't pay off when

  • Prompts are unique, or the state cannot be serialized and restored exactly, so nothing is reused and saving it is wasted work.
  • Restoring costs more than recomputing, for example a short prefix behind a slow disk; the plan then recomputes.
  • The prefix cannot be matched safely, for example when a chat template renders the previous reply differently on the next turn.

What must stay true

  • A bundle is reused only for exactly the same prefix tokens, weights, state layout and codec. Similar text or matching tensor names are not enough.
  • Only committed state is saved, and a bundle is published or restored whole, never one component alone.
  • Every source, destination and lease stays alive until each copy has really finished.