crit-hub

device-expert-cache

Expert caching

A mixture-of-experts model keeps its hot experts in GPU memory and fetches the rest from host memory on demand.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A mixture-of-experts model has far more expert weights than one token uses, often more than the GPU can hold.
  • Each token only needs a few experts per layer, but which ones changes from token to token.

How it works

  1. The router picks a few experts for the token in this layer.
  2. Experts already in a GPU slot are hits. They are pinned, so nothing this layer reads can be evicted.
  3. A miss needs a slot. A clock hand sweeps the slots, clears recently used ones and takes the first it finds unused.
  4. The missing expert is copied from host memory into that slot over PCIe, and the layer runs once every copy has landed.
  5. The plan runs on the GPU itself, so a whole decode step needs no round trip to the CPU.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
policyclockclock, rankedclock: admit misses as they come (admit_quota, second_touch); ranked: swap the experts with the most decayed routes in at window boundaries (decay, period, max_swaps, burst_swaps)
slots64int, at least 1Device expert slots per layer
layers128int, at least 1Layers the cache plans
experts512int, at least 1Experts per layer
admit_quota-1int, at least -1Misses a step may admit, with clock (-1: no limit)
second_touchfalsetrue, falseAdmit an expert only on its second miss, with clock
decay0.85float 0 to 1How fast a route's weight fades, with ranked
period3int, at least 1Steps per window between swaps, with ranked
max_swaps96int, at least 0Experts swapped in per window, with ranked
burst_swaps384int, at least 0Swaps allowed in a burst after a shift in routing
cpu_tierfalsetrue, falsea CPU half computes the misses the device form does not admit (slot -2): needed for admit_quota, quota_table, second_touch, and for ranked to admit nothing on the step's path
quota_table""strdevice form: the admission quota by a plan's miss count, comma-separated from 0 misses (the last serves larger counts; split.py quota_table), over admit_quota; per token bucket as "tokens:q0,q1;tokens:q0,..." (a plan of t tokens reads the largest bucket <= t)
ringborrowborrow, fixeddevice form: the prompt ring borrowed from the cache's own slots while prompts run (back after a round of decode plans without a prompt), or kept apart
ring_slots-1int, at least -1device form: slots of the prompt ring (-1: twice the experts of a layer, at most half the own slots); at least the experts of a layer for long prompt chunks
stage_routes0int, at least 0device form: a prompt chunk of at least this many routes stages every non-resident expert of a layer by the copy engine (0: as many as the experts of a layer); shorter ones copy their routed misses on the device
fill_blocks0int, at least 0device form: thread blocks of the fill (0: half the multiprocessors; a CSF store's program fill: twice)
fill_vector164, 8, 16device form: bytes a fill thread loads at a time
nativetruetrue, falsedevice form: run the directory's CUDA plan and fill (built at load); false: its torch ops
round_rows64int, at least 1device form: the rows a round may have (the per-row reads and hits counters count_round adds into the round)
configuration.json
{"plugins": {
  "use":     {"expert_residency": "device-expert-cache"},
  "options": {"device-expert-cache": {"policy": "clock", "slots": 64, "layers": 128}}
}}

Pays off when

  • The model is larger than GPU memory, and routing keeps returning to the same experts.
  • Decoding, where each step touches few experts.

Doesn't pay off when

  • Every expert fits on the GPU already.
  • Large prefill batches touch most experts of a layer anyway, so a plan grouped by expert on the host is simpler.

What must stay true

  • An expert maps to at most one slot, and its copy completes before any arithmetic reads the slot.
  • Every expert a plan reads is pinned against that plan's own evictions.
  • The cache keeps adapting while it decodes; it is not frozen after a warmup.