partitioned-attention
Partitioned attention
A long sequence's keys and values are split across workers so none holds them all; each attends over its share, and the partials merge exactly by their softmax totals.
The problem
- The KV cache, the keys and values of every past token, grows with the context. At long contexts one GPU cannot hold it, and a copy on every GPU wastes memory.
- Attention takes one softmax over all keys at once, so outputs computed over separate parts of the sequence cannot simply be added together.
How it works
- Position p of the sequence goes to worker p mod N (the remainder of p divided by N), at slot p div N there, so the shares never differ by more than one key as the sequence grows.
- Every worker gets the same query, scale and mask. The causal mask compares global positions, so a key after the query is hidden wherever it is stored.
- Each worker takes a softmax over its own visible keys and sends back two things, its partial output and L, the log of its softmax total.
- The merge takes the largest L as M and weights each partial by exp(L minus M), divided by the sum of those weights. Subtracting M keeps every exp from overflowing.
- The weighted partials add up to what one softmax over all the keys gives, up to rounding. A worker with no visible keys sends a zero output with L of minus infinity, so its weight is zero.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
world | 2 | int 1 to 64 | Ranks of the host form's interleaved ownership (position mod world) |
page_size | 256 | int 1 to 65536 | Positions per local page of the host form's ranks |
partitions | 0 | int 0 to 1024 | KV partitions of the device form's split-KV reader (0: from the GPU's SM count, the KV heads and the page kind); a row's bits depend on it |
piece | 0 | int 0 to 128 | Positions per piece of the split-KV reader (0: the kernel's default from the head width and the page kind); a row's bits depend on it |
kernels | auto | auto, torch | auto: the registry's paged-kv-attention kernels where they load (FlashInfer for 16-bit and FP8 prompt rows); torch: every row in torch |
workspace_mib | 128 | int 1 to 65536 | MiB of partials the split-KV reader allocates when the plugin is made (memory() declares them): a launch reads as many rows as they hold |
flashinfer | auto | auto, off | auto: FlashInfer's paged prefill for prompt rows of 16-bit and FP8 pages where it is installed, its workspaces allocated when the plugin is made; off: none (a model whose KV pages FlashInfer cannot read, fp8:row or INT8, allocates nothing for it) |
flashinfer_mib | 128 | int 16 to 65536 | MiB of FlashInfer's scheduling workspace, allocated when the plugin is made where FlashInfer serves (memory() declares it with its wrappers' own workspaces) |
prompt_reader | prefill | prefill, decode | prefill: prompt rows through a prompt reader (FlashInfer's prefill or the registry's), fastest, another reduction than decode rows' (a scored token may differ from the served one by more than 0.001 nats); decode: prompt rows (a step's and a captured chunk's) in the decode's reduction, the split-KV reader as window rows of up to 64 columns (each program folding its partitions as the merge does) or its torch fallback over the page tables: a scored token equals the same token served, bit for bit, over every page type (window rows through the split-KV reader too, no FlashInfer) |
configuration.json
{"plugins": {
"use": {"attention": "partitioned-attention"},
"options": {"partitioned-attention": {"world": 2, "page_size": 256, "partitions": 0}}
}}Pays off when
- A long context's KV cache does not fit on one GPU, or keeping a copy on every GPU wastes memory.
- Dense attention, or sparse attention whose keys are chosen over the whole sequence before it is split.
Doesn't pay off when
- Short contexts, where exchanging the partials between workers costs more than the memory it saves.
- The number of workers changes often, since each change remaps the live KV cache and invalidates captured graphs.
What must stay true
- Every visible key is owned by exactly one worker and counted once. A missing or doubled key gives a plausible but wrong answer.
- Causality uses global positions, and every worker applies the same query, scale and choice of keys.
- A missing or failed partial blocks the output; it is never dropped from the merge.
- No visible key on any worker, a NaN, or an L of plus infinity is an error, never a zero output.
crit-hub