crit-hub

kv_cache

The keys and values of every sequence: pages in a format (BF16, FP8, INT8), forks for shared prefixes, staged rows a speculative round commits or rolls back.

required

In crit's library

Interface

The methods the engine calls on the slot's plugin, in their host form: token ids, rows of numbers and dicts. Every plugin of the slot has them, which is what lets an ablation switch one for another.

MethodWhat it does
open, releaseA sequence's pages
stage, commit, rollbackA round's rows, kept or dropped
rows, forkA sequence's rows; a shared prefix

Device form: pool(layers, kv_heads, head_dim, tokens, device, dtype): a KV pool with open, fork, reserve, write, keep and gather