attention
Each token's queries over its sequence's keys and values, up to its position, read from the KV cache's pages.
required
In crit's library
Softmax attentionExact causal softmax attention: each row's queries over its sequence's pages, read in place by the kernel that fits the page format.attentionPartitioned attentionA long sequence's keys and values are split across workers so none holds them all; each attends over its share, and the partials merge exactly by their softmax totals.attention
Interface
The methods the engine calls on the slot's plugin, in their host form: token ids, rows of numbers and dicts. Every plugin of the slot has them, which is what lets an ablation switch one for another.
| Method | What it does |
|---|---|
attend(...) | Each row's queries over its sequence's keys and values |
Device form: forward(layer, queries, pool, batch, scale): each token attends its sequence's KV up to its position
crit-hub