crit-hub

attention-state

Paged KV cache with prefix sharing

The KV cache is cut into fixed pages, so requests that start with the same prompt point at the same pages instead of computing and storing them again.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Every token of the context keeps its keys and values on the GPU, and long contexts fill memory fast.
  • Chats, agents and follow-up turns resend the same system prompt and history, and prefill computes it again each time.

How it works

  1. The cache is a pool of pages of a fixed number of tokens. Each request has a block table that lists its pages in order.
  2. A new request whose prompt starts like a cached one points its block table at the same pages and skips their prefill.
  3. Shared pages are read only. A request that writes into a shared, partly filled page first copies it (copy on write).
  4. When a request ends, its private pages return to the pool; shared pages stay while any request still uses them.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
page16int 1 to 4096Tokens per KV page
step_rows0int 0 to 256Window rows (decode tokens, verify windows) a step may hold in the device pool's step buffers, at fixed addresses for CUDA graphs (0: no step buffers)
step_tokens8192int 1 to 1048576Tokens a step may hold with step buffers
window16int 1 to 64Most tokens of a row read as a window row (decode tokens, verify windows, short prompt pieces: the attention plugins' split-KV reader, and the step buffers' rows); a longer row is a prompt row
round_rows8int 0 to 256Round slots a captured round may bind sequences to (rounds:kv_cache)
fresh_mib0int 0 to 4096MiB of 16-bit keys and values of a prompt chunk, made when the plugin is made on a CUDA engine (memory() declares them): over FP8 and INT8 pages a prompt row (more than window tokens) attends its own chunk as the projections gave it and the earlier positions from the pages (a step past the buffer, or 0, reads its own chunk from the pages). Off by default: fresh reads make a scored continuation part from the served decode by up to the pages' quantization (a 27B engine's served and scored alternatives 3.2 nats apart, past speed's 2.0 honesty bound), so an engine turns it on only where its honesty bound and quality measured with it allow
kernelsautoauto, torchauto: pages written by the registry's paged-kv-attention kernels where they load; torch: in torch
configuration.json
{"plugins": {
  "use":     {"kv_cache": "attention-state"},
  "options": {"attention-state": {"page": 16, "step_rows": 0, "step_tokens": 8192}}
}}

Pays off when

  • Many requests share a system prompt, documents or chat history.
  • Follow-up turns, where only the new message needs prefill.

Doesn't pay off when

  • Requests share nothing, so pages add bookkeeping and no reuse.
  • The cached state cannot be matched exactly, for example after a change of model, KV encoding or sampling state.

What must stay true

  • Only committed tokens are shared; a speculative or rejected row never lands in a shared page.
  • A shared page is never written; a page is reused only after every read of it has finished.
  • A prefix matches on its exact tokens and on everything that shaped its state.