kv_cache
The keys and values of every sequence: pages in a format (BF16, FP8, INT8), forks for shared prefixes, staged rows a speculative round commits or rolls back.
required
In crit's library
Interface
The methods the engine calls on the slot's plugin, in their host form: token ids, rows of numbers and dicts. Every plugin of the slot has them, which is what lets an ablation switch one for another.
| Method | What it does |
|---|---|
open, release | A sequence's pages |
stage, commit, rollback | A round's rows, kept or dropped |
rows, fork | A sequence's rows; a shared prefix |
Device form: pool(layers, kv_heads, head_dim, tokens, device, dtype): a KV pool with open, fork, reserve, write, keep and gather
crit-hub