expert_residency
Which experts stay on the device and which are fetched or computed elsewhere, under a cache policy.
optionalhost policy
In crit's library
Expert cachingA mixture-of-experts model keeps its hot experts in GPU memory and fetches the rest from host memory on demand.expert_residencyExpert residencyAn expert cache that lets readers see only complete expert bundles, fills during decode but not prefill, and borrows memory the KV cache and prompt workspace are not using.expert_residency
Interface
The methods the engine calls on the slot's plugin, in their host form: token ids, rows of numbers and dicts. Every plugin of the slot has them, which is what lets an ablation switch one for another.
| Method | What it does |
|---|---|
plan(...) | Which experts the device holds |
resident(...) | Whether an expert is on the device now |
crit-hub