crit-hub

host-expert-tier

CPU expert tier

A layer's share of routed experts runs on the CPU at FP32, with the device's integer rows and its plan order, so the sum is the one the GPU would give.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • When a mixture of experts does not fit on the GPU, an expert it misses is either copied over the link or computed on the CPU from host memory. A desktop reads host memory at about twice its PCIe link's rate.
  • An expert computed on the CPU in another order than the GPU gives other bits, so which side ran it would change the text.

How it works

  1. The GPU's expert cache decides which misses go to the CPU and posts their routes into a mailbox slot: expert ids, each route's quantized input and room for the outputs.
  2. A pool of workers, one per physical core less the caller's, groups the routes by expert, up to max_tokens tokens a chunk, so each expert's rows are read once for all its tokens.
  3. Each chunk runs the gate/up rows in the GPU's integer block order, the declared SwiGLU and the intermediate's declared quantizer, then the down rows.
  4. Each route gets its own unweighted output row. The GPU multiplies by the routing weight and sums a token's routes in its own plan order.
  5. The order of that sum never depends on which side ran an expert, so the CPU's share never changes a token.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
max_tokens8int, at least 1host form: routes of one expert in a chunk
nativeautoauto, on, offthe device form's rows: the native tier built at load (on: refuse to load without it), or the declared order in torch (off), the same bits; auto takes the native tier where it builds
workers-1int, at least -1the native tier's worker threads (-1: one per physical core but the caller's)
pintruetrue, falsepin each worker to a physical core of its own (Linux)
spin_us100int, at least 0the least an idle CPU worker spins before it sleeps
rows_per_task128int, at least 64rows of one expert in a pool task
spin_max_us3000int, at least 0an idle CPU worker spins as long as the recent gaps between batches (1.5 times the longest of the last 16), at most this, then sleeps: awake through a decode step's layers, asleep between steps (0: spin_us only). The default was measured on one machine (6 cores, WSL2, decode layers about 1.2 ms apart): measure your own cadence
configuration.json
{"plugins": {
  "use":     {"experts": "host-expert-tier"},
  "options": {"host-expert-tier": {"max_tokens": 8, "native": "auto", "workers": -1}}
}}

Pays off when

  • The experts do not all fit on the GPU, and the probed CPU lane beats copying misses over the link, alone or beside the copies.
  • It runs beside the GPU's expert cache, which sends it the misses.

Doesn't pay off when

  • Every expert fits on the device.
  • The probe and a measured layer give the CPU no share.

What must stay true

  • One declared arithmetic on every processor: the integer rows, the declared SwiGLU and the intermediate's declared quantizer.
  • Rows go back unweighted, one per route, and the GPU sums them in plan order.
  • A worker claims a task with a word that carries the batch's epoch, so a worker still leaving one batch claims nothing of the next.
  • One core is left to the caller.