softmax-attention
Softmax attention
Exact causal softmax attention: each row's queries over its sequence's pages, read in place by the kernel that fits the page format.
The problem
- Each token attends to every token before it: it scores its query against their keys, takes a softmax, and sums their values with those weights.
- Faster attention (split across workers, low-bit pages, fused kernels) must give what that plain softmax gives.
How it works
- Each query sees the keys and values at positions up to its own, read from the KV cache's rows or pages.
- Its scores against those keys, times the scale, go through one softmax, and the output is the values weighted by it.
- On the device, 16-bit and FP8 pages are read in place by FlashInfer's paged kernels when FlashInfer is installed, with the split of the KV pinned so a row does not depend on its neighbours.
- INT8 pages are read by attention-state's int8 reader. Otherwise each row's keys and values are gathered from the pool and run through torch's scaled dot product attention, the plain path the others are tested against.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
kernels | auto | auto, torch | auto: FlashInfer for 16-bit and FP8 pages when it is installed, the registry's paged-kv-attention kernels where they load; torch: every row gathered and multiplied in torch |
int8_qk | false | true, false | over INT8 pages, prompt rows' queries quantized to INT8 per token and head and multiplied with the INT8 keys on integer tensor cores (twice the BF16 rate, an approximation of the queries) |
decode | split | split, flashinfer | a step's window rows (decode tokens, verify windows): split: the registry's split-KV reader wherever it loads, the same kernel as captured rounds (a decode row's bits the same in a step and in a round: a plain or log-probability request's tokens equal a drafted round's); flashinfer: FlashInfer's decode and prefill over 16-bit and FP8 pages where it is installed, else the split-KV reader (another reduction than a captured round's, so a row a step decodes may differ from the same row in a round: for engines that decode in steps only) |
prompt_reader | prefill | prefill, decode | prefill: prompt rows through a prompt reader (FlashInfer's prefill or the registry's), fastest, another reduction than decode rows' (a scored token may differ from the served one by more than 0.001 nats); decode: prompt rows (a step's and a captured chunk's) in the decode's reduction, the split-KV reader as window rows of up to 64 columns (each program folding its partitions as the merge does) or its torch fallback over the page tables: a scored token equals the same token served, bit for bit, over every page type (window rows through the split-KV reader too, no FlashInfer) |
partitions | 0 | int 0 to 1024 | KV partitions of the split-KV reader (0: from the GPU's SM count, the KV heads and the page kind); a row's bits depend on it |
piece | 0 | int 0 to 128 | Positions per piece of the split-KV reader (0: the kernel's default from the head width and the page kind); a row's bits depend on it |
workspace_mib | 128 | int 1 to 65536 | MiB of partials the split-KV reader allocates when the plugin is made (memory() declares them): a launch reads as many rows as they hold |
flashinfer | auto | auto, off | auto: FlashInfer's paged kernels where it is installed, for pools of 16-bit and FP8 pages, its workspaces allocated when the plugin is made; off: none (a model whose KV pages FlashInfer cannot read, fp8:row or INT8, allocates nothing for it) |
flashinfer_mib | 128 | int 16 to 65536 | MiB of FlashInfer's scheduling workspace, allocated when the plugin is made where FlashInfer serves (memory() declares it with its wrappers' own workspaces) |
configuration.json
{"plugins": {
"use": {"attention": "softmax-attention"},
"options": {"softmax-attention": {"kernels": "auto", "int8_qk": false, "decode": "split"}}
}}Pays off when
- The model has attention layers: it is the plain attention, and with FlashInfer it serves at speed.
- An attention plugin is checked:
partitioned-attentionis compared with it.
Doesn't pay off when
- One device cannot hold a long sequence's keys and values.
partitioned-attentionsplits them across workers. - FlashInfer is not installed, so 16-bit and FP8 pages fall back to gathered rows and torch, which is slower.
What must stay true
- A query attends only to positions up to its own.
- A row's output does not depend on the rows computed beside it.
crit-hub