crit-hub

expert-residency

Expert residency

An expert cache that lets readers see only complete expert bundles, fills during decode but not prefill, and borrows memory the KV cache and prompt workspace are not using.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • An expert is several tensors, such as its 4-bit codes and their scales. A reader that sees the new codes next to the old scales computes with half an expert.
  • A long prompt touches nearly every expert once. If it could fill the cache, it would evict the experts decode keeps coming back to.
  • The KV cache reserves memory for the longest context, and the prompt workspace sits idle between prompts, while the expert cache is short of slots.

How it works

  1. A bank planner sizes the cache from a byte budget. Each slot holds one complete bundle, meaning every tensor of one expert with its padding, so the slot count is what really fits.
  2. A miss starts an upload. The slot turns LOADING under a new generation (a counter that changes whenever its content is replaced), and every tensor of the bundle is copied on its own.
  3. The slot is published READY only when every copy reports success. A failed copy leaves it EMPTY, because its old expert is already partly overwritten.
  4. Admission depends on the phase. Decode populates the cache; prefill only reuses what is already there and runs its misses from host memory, so it uploads and evicts nothing.
  5. Idle KV positions and the prompt workspace lend slots to the cache. Before the owner writes there, it takes a slot loan. Readers must settle first, survivors can move to free slots, and the slots come back empty.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
slots64int, at least 1Expert bundles the cache holds
expert_bytes1int, at least 1Bytes of one expert bundle
revisionmodelstrThe model revision bundles are keyed by
encodingbundlestrHow a bundle is encoded
configuration.json
{"plugins": {
  "use":     {"expert_residency": "expert-residency"},
  "options": {"expert-residency": {"slots": 64, "expert_bytes": 1, "revision": "model"}}
}}

Pays off when

  • The routed experts do not fit on the GPU, and decode keeps returning to the same ones.
  • The engine reserves KV memory for long contexts that most requests never fill, or a prompt workspace that is idle while it decodes.
  • Prompts and decode share one cache, and prefill must not wash out the experts decode relies on.

Doesn't pay off when

  • Every expert already fits on the GPU, or routing rarely repeats, so uploads and bookkeeping cost more than the hits save.
  • Requests fill their whole context, so the KV cache takes its lent slots back almost as soon as the cache gains them.
  • A bigger expert pool takes memory that KV state or scratch needed more.

What must stay true

  • A slot is readable only after every tensor of its bundle has landed, and only copy receipts for its current generation count.
  • A reader keeps its slot pinned until its reads have physically finished. No upload and no loan overwrites a pinned slot.
  • A failed or aborted upload leaves the slot empty. The old expert is never relabelled back from partly overwritten bytes.
  • Lent memory stays charged once, to its owner. The owner writes only after the cache's readers have settled, and the slots return from a loan empty.