crit-hub

partitioned-attention

Partitioned attention

A long sequence's keys and values are split across workers so none holds them all; each attends over its share, and the partials merge exactly by their softmax totals.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • The KV cache, the keys and values of every past token, grows with the context. At long contexts one GPU cannot hold it, and a copy on every GPU wastes memory.
  • Attention takes one softmax over all keys at once, so outputs computed over separate parts of the sequence cannot simply be added together.

How it works

  1. Position p of the sequence goes to worker p mod N (the remainder of p divided by N), at slot p div N there, so the shares never differ by more than one key as the sequence grows.
  2. Every worker gets the same query, scale and mask. The causal mask compares global positions, so a key after the query is hidden wherever it is stored.
  3. Each worker takes a softmax over its own visible keys and sends back two things, its partial output and L, the log of its softmax total.
  4. The merge takes the largest L as M and weights each partial by exp(L minus M), divided by the sum of those weights. Subtracting M keeps every exp from overflowing.
  5. The weighted partials add up to what one softmax over all the keys gives, up to rounding. A worker with no visible keys sends a zero output with L of minus infinity, so its weight is zero.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
world2int 1 to 64Ranks of the host form's interleaved ownership (position mod world)
page_size256int 1 to 65536Positions per local page of the host form's ranks
partitions0int 0 to 1024KV partitions of the device form's split-KV reader (0: from the GPU's SM count, the KV heads and the page kind); a row's bits depend on it
piece0int 0 to 128Positions per piece of the split-KV reader (0: the kernel's default from the head width and the page kind); a row's bits depend on it
kernelsautoauto, torchauto: the registry's paged-kv-attention kernels where they load (FlashInfer for 16-bit and FP8 prompt rows); torch: every row in torch
workspace_mib128int 1 to 65536MiB of partials the split-KV reader allocates when the plugin is made (memory() declares them): a launch reads as many rows as they hold
flashinferautoauto, offauto: FlashInfer's paged prefill for prompt rows of 16-bit and FP8 pages where it is installed, its workspaces allocated when the plugin is made; off: none (a model whose KV pages FlashInfer cannot read, fp8:row or INT8, allocates nothing for it)
flashinfer_mib128int 16 to 65536MiB of FlashInfer's scheduling workspace, allocated when the plugin is made where FlashInfer serves (memory() declares it with its wrappers' own workspaces)
prompt_readerprefillprefill, decodeprefill: prompt rows through a prompt reader (FlashInfer's prefill or the registry's), fastest, another reduction than decode rows' (a scored token may differ from the served one by more than 0.001 nats); decode: prompt rows (a step's and a captured chunk's) in the decode's reduction, the split-KV reader as window rows of up to 64 columns (each program folding its partitions as the merge does) or its torch fallback over the page tables: a scored token equals the same token served, bit for bit, over every page type (window rows through the split-KV reader too, no FlashInfer)
configuration.json
{"plugins": {
  "use":     {"attention": "partitioned-attention"},
  "options": {"partitioned-attention": {"world": 2, "page_size": 256, "partitions": 0}}
}}

Pays off when

  • A long context's KV cache does not fit on one GPU, or keeping a copy on every GPU wastes memory.
  • Dense attention, or sparse attention whose keys are chosen over the whole sequence before it is split.

Doesn't pay off when

  • Short contexts, where exchanging the partials between workers costs more than the memory it saves.
  • The number of workers changes often, since each change remaps the live KV cache and invalidates captured graphs.

What must stay true

  • Every visible key is owned by exactly one worker and counted once. A missing or doubled key gives a plausible but wrong answer.
  • Causality uses global positions, and every worker applies the same query, scale and choice of keys.
  • A missing or failed partial blocks the output; it is never dropped from the merge.
  • No visible key on any worker, a NaN, or an L of plus infinity is an error, never a zero output.