heterogeneous-expert-dispatch
Splitting experts between CPU and GPU
For the experts the GPU does not hold, each step picks between copying them over and computing them on the CPU, choosing the split with the lowest measured time.
The problem
- When the experts do not fit on the GPU, a missed expert is either copied over PCIe or computed on the CPU from host memory.
- Which is faster depends on the machine, the expert's size and how many tokens use it, so a fixed CPU share tuned on one machine is wrong on the next.
- Models route differently (softmax or sigmoid, a selection bias, expert groups, renormalized weights), and the GPU kernel and every host check must apply the same rule.
How it works
- One Routing description says how a token's router logits become experts and weights, one route per pick. The GPU kernel implements it, and a host reference checks it on the model's settings.
- Experts already in the GPU cache stay there, protected for the step. The misses are put in a fixed order.
- Every split is priced, with the first 0, 1, 2 or more misses on the CPU and the rest uploaded. Times come from measurements on this machine at exactly those counts, and since the CPU and the GPU work at the same time, the slower side sets the estimate.
- The lowest estimate wins. Each route then names its source, which is a cache slot, a scratch slot that receives an uploaded miss, or the CPU.
- Contributions are weighted by their routes and summed in a fixed order. With the CPU computing in the GPU's exact operation order, the result does not depend on whether the CPU or the GPU ran an expert.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
device_slots | 64 | int, at least 0 | Expert slots on the device; the other experts run on the host |
resident | -1 | int, at least -1 | forward() over the plugin's own store: the experts of each layer kept on the device (the lowest ids; -1 all); the others go to the CPU tier |
max_tokens | 64 | int, at least 1 | the most tokens one post sends the CPU (a decode step or verify round; the mailbox holds this many tokens of the model's top routes); a longer call posts in pieces |
join | auto | auto, device_poll, stream_wait, host_eager | how the stream waits for the CPU: auto takes the cheaper capturable join measured at load (host-work-mailbox measure_joins) |
native | auto | auto, on, off | the CPU tier's rows: native (built at load) or the declared order in torch; without the native tier CPU routes are joined on the host, outside capture |
workers | -1 | int, at least -1 | the CPU tier's worker threads (-1: one per physical core but the caller's) |
pin | true | true, false | pin each CPU worker to a physical core of its own (Linux) |
spin_us | 100 | int, at least 0 | the least an idle CPU worker spins before it sleeps |
spin_max_us | 3000 | int, at least 0 | an idle CPU worker spins as long as the recent gaps between batches (1.5 times the longest of the last 16), at most this, then sleeps: awake through a decode step's layers, asleep between steps (0: spin_us only). The default was measured on one machine (6 cores, WSL2, decode layers about 1.2 ms apart): measure your own cadence |
rows_per_task | 128 | int, at least 64 | rows of one expert in a CPU pool task |
idle_after_us | 2000 | int, at least 0 | the mailbox's service thread polls the doorbells without pause this long after a job, then sleeps idle_sleep_us between polls: measured on one machine with decode layers about 1.2 ms apart |
idle_sleep_us | 50 | int, at least 0 | the service thread's sleep between polls when idle (the first job after a pause waits up to this) |
configuration.json
{"plugins": {
"use": {"experts": "heterogeneous-expert-dispatch"},
"options": {"heterogeneous-expert-dispatch": {"device_slots": 64, "resident": -1, "max_tokens": 64}}
}}Pays off when
- Host memory reads faster than the link copies to the GPU, as on many desktop machines, so the CPU is a real source of misses.
- Decode and short prompt chunks, where each step misses a handful of experts.
Doesn't pay off when
- Every expert fits on the GPU, or the CPU computes an expert slower than its copy takes, so measurement gives the CPU no share.
- Host memory is already busy, since the CPU's misses read the same memory as the copies and its threads compete with the thread that launches GPU work.
- Routing keeps returning to the same experts. A miss the CPU computes is not admitted to the GPU cache, so the miss rate rises with the CPU share.
What must stay true
- Every route keeps its expert, position and weight. A missing expert is never swapped for a resident one, and a repeated expert is uploaded once but counted at every occurrence.
- A split with no measured time can never win, and timings are thrown away when the model, layout, code or machine changes.
- Cache hits are protected before scratch is chosen, scratch never overlaps a ready slot, and an upload lands completely before the GPU reads it.
- The CPU computes an expert in the GPU kernel's exact operation order, so the split never changes the output.
crit-hub