device-expert-cache
Expert caching
A mixture-of-experts model keeps its hot experts in GPU memory and fetches the rest from host memory on demand.
The problem
- A mixture-of-experts model has far more expert weights than one token uses, often more than the GPU can hold.
- Each token only needs a few experts per layer, but which ones changes from token to token.
How it works
- The router picks a few experts for the token in this layer.
- Experts already in a GPU slot are hits. They are pinned, so nothing this layer reads can be evicted.
- A miss needs a slot. A clock hand sweeps the slots, clears recently used ones and takes the first it finds unused.
- The missing expert is copied from host memory into that slot over PCIe, and the layer runs once every copy has landed.
- The plan runs on the GPU itself, so a whole decode step needs no round trip to the CPU.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
policy | clock | clock, ranked | clock: admit misses as they come (admit_quota, second_touch); ranked: swap the experts with the most decayed routes in at window boundaries (decay, period, max_swaps, burst_swaps) |
slots | 64 | int, at least 1 | Device expert slots per layer |
layers | 128 | int, at least 1 | Layers the cache plans |
experts | 512 | int, at least 1 | Experts per layer |
admit_quota | -1 | int, at least -1 | Misses a step may admit, with clock (-1: no limit) |
second_touch | false | true, false | Admit an expert only on its second miss, with clock |
decay | 0.85 | float 0 to 1 | How fast a route's weight fades, with ranked |
period | 3 | int, at least 1 | Steps per window between swaps, with ranked |
max_swaps | 96 | int, at least 0 | Experts swapped in per window, with ranked |
burst_swaps | 384 | int, at least 0 | Swaps allowed in a burst after a shift in routing |
cpu_tier | false | true, false | a CPU half computes the misses the device form does not admit (slot -2): needed for admit_quota, quota_table, second_touch, and for ranked to admit nothing on the step's path |
quota_table | "" | str | device form: the admission quota by a plan's miss count, comma-separated from 0 misses (the last serves larger counts; split.py quota_table), over admit_quota; per token bucket as "tokens:q0,q1;tokens:q0,..." (a plan of t tokens reads the largest bucket <= t) |
ring | borrow | borrow, fixed | device form: the prompt ring borrowed from the cache's own slots while prompts run (back after a round of decode plans without a prompt), or kept apart |
ring_slots | -1 | int, at least -1 | device form: slots of the prompt ring (-1: twice the experts of a layer, at most half the own slots); at least the experts of a layer for long prompt chunks |
stage_routes | 0 | int, at least 0 | device form: a prompt chunk of at least this many routes stages every non-resident expert of a layer by the copy engine (0: as many as the experts of a layer); shorter ones copy their routed misses on the device |
fill_blocks | 0 | int, at least 0 | device form: thread blocks of the fill (0: half the multiprocessors; a CSF store's program fill: twice) |
fill_vector | 16 | 4, 8, 16 | device form: bytes a fill thread loads at a time |
native | true | true, false | device form: run the directory's CUDA plan and fill (built at load); false: its torch ops |
round_rows | 64 | int, at least 1 | device form: the rows a round may have (the per-row reads and hits counters count_round adds into the round) |
configuration.json
{"plugins": {
"use": {"expert_residency": "device-expert-cache"},
"options": {"device-expert-cache": {"policy": "clock", "slots": 64, "layers": 128}}
}}Pays off when
- The model is larger than GPU memory, and routing keeps returning to the same experts.
- Decoding, where each step touches few experts.
Doesn't pay off when
- Every expert fits on the GPU already.
- Large prefill batches touch most experts of a layer anyway, so a plan grouped by expert on the host is simpler.
What must stay true
- An expert maps to at most one slot, and its copy completes before any arithmetic reads the slot.
- Every expert a plan reads is pinned against that plan's own evictions.
- The cache keeps adapting while it decodes; it is not frozen after a warmup.
crit-hub