crit-hub

heterogeneous-expert-dispatch

Splitting experts between CPU and GPU

For the experts the GPU does not hold, each step picks between copying them over and computing them on the CPU, choosing the split with the lowest measured time.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • When the experts do not fit on the GPU, a missed expert is either copied over PCIe or computed on the CPU from host memory.
  • Which is faster depends on the machine, the expert's size and how many tokens use it, so a fixed CPU share tuned on one machine is wrong on the next.
  • Models route differently (softmax or sigmoid, a selection bias, expert groups, renormalized weights), and the GPU kernel and every host check must apply the same rule.

How it works

  1. One Routing description says how a token's router logits become experts and weights, one route per pick. The GPU kernel implements it, and a host reference checks it on the model's settings.
  2. Experts already in the GPU cache stay there, protected for the step. The misses are put in a fixed order.
  3. Every split is priced, with the first 0, 1, 2 or more misses on the CPU and the rest uploaded. Times come from measurements on this machine at exactly those counts, and since the CPU and the GPU work at the same time, the slower side sets the estimate.
  4. The lowest estimate wins. Each route then names its source, which is a cache slot, a scratch slot that receives an uploaded miss, or the CPU.
  5. Contributions are weighted by their routes and summed in a fixed order. With the CPU computing in the GPU's exact operation order, the result does not depend on whether the CPU or the GPU ran an expert.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
device_slots64int, at least 0Expert slots on the device; the other experts run on the host
resident-1int, at least -1forward() over the plugin's own store: the experts of each layer kept on the device (the lowest ids; -1 all); the others go to the CPU tier
max_tokens64int, at least 1the most tokens one post sends the CPU (a decode step or verify round; the mailbox holds this many tokens of the model's top routes); a longer call posts in pieces
joinautoauto, device_poll, stream_wait, host_eagerhow the stream waits for the CPU: auto takes the cheaper capturable join measured at load (host-work-mailbox measure_joins)
nativeautoauto, on, offthe CPU tier's rows: native (built at load) or the declared order in torch; without the native tier CPU routes are joined on the host, outside capture
workers-1int, at least -1the CPU tier's worker threads (-1: one per physical core but the caller's)
pintruetrue, falsepin each CPU worker to a physical core of its own (Linux)
spin_us100int, at least 0the least an idle CPU worker spins before it sleeps
spin_max_us3000int, at least 0an idle CPU worker spins as long as the recent gaps between batches (1.5 times the longest of the last 16), at most this, then sleeps: awake through a decode step's layers, asleep between steps (0: spin_us only). The default was measured on one machine (6 cores, WSL2, decode layers about 1.2 ms apart): measure your own cadence
rows_per_task128int, at least 64rows of one expert in a CPU pool task
idle_after_us2000int, at least 0the mailbox's service thread polls the doorbells without pause this long after a job, then sleeps idle_sleep_us between polls: measured on one machine with decode layers about 1.2 ms apart
idle_sleep_us50int, at least 0the service thread's sleep between polls when idle (the first job after a pause waits up to this)
configuration.json
{"plugins": {
  "use":     {"experts": "heterogeneous-expert-dispatch"},
  "options": {"heterogeneous-expert-dispatch": {"device_slots": 64, "resident": -1, "max_tokens": 64}}
}}

Pays off when

  • Host memory reads faster than the link copies to the GPU, as on many desktop machines, so the CPU is a real source of misses.
  • Decode and short prompt chunks, where each step misses a handful of experts.

Doesn't pay off when

  • Every expert fits on the GPU, or the CPU computes an expert slower than its copy takes, so measurement gives the CPU no share.
  • Host memory is already busy, since the CPU's misses read the same memory as the copies and its threads compete with the thread that launches GPU work.
  • Routing keeps returning to the same experts. A miss the CPU computes is not admitted to the GPU cache, so the miss rate rises with the CPU share.

What must stay true

  • Every route keeps its expert, position and weight. A missing expert is never swapped for a resident one, and a repeated expert is uploaded once but counted at every occurrence.
  • A split with no measured time can never win, and timings are thrown away when the model, layout, code or machine changes.
  • Cache hits are protected before scratch is chosen, scratch never overlaps a ready slot, and an upload lands completely before the GPU reads it.
  • The CPU computes an expert in the GPU kernel's exact operation order, so the split never changes the output.