Joining expert contributions
Experts Expert outputs from the CPU and GPU arrive at different times; they are summed only once all have settled, in a fixed order, so arrival order never changes the result.
In an engine
No slot of its own: it lends its code to heterogeneous-expert-dispatch.
The problem
- When some of a token's experts run on the CPU and others on the GPU, their outputs arrive at different times, in a different order from run to run.
- Floating-point addition rounds after every step, so a sum taken in arrival order can change in its last digits from one run to the next.
- Releasing an expert's weights, or publishing the output, while another expert is still running would let that expert read freed memory or expose a half-finished result.
How it works
- The token's routing has one position per chosen expert, and each position owes exactly one contribution, already multiplied by its routing weight.
- Each position's work is launched on the CPU or the GPU and handed to the join with a receipt, which reports when the work is done, and its source, the weights it reads. The join keeps every source alive.
- Contributions finish in any order. The join is ready only when every position has been handed in and every receipt reports done.
- The join then adds the contributions in position order, never in arrival order. Only now are the sources released and the sum published.
- If a contribution failed, the join still waits for all the others to finish, then raises the error and publishes nothing.
Pays off when
- One routed operation launches independent work on several sources, such as CPU and GPU experts, and must produce one result.
- Runs must give the same output whatever the timing, so the sum cannot depend on which source was faster.
Doesn't pay off when
- One fused operation already produces the whole result, so there is nothing to join.
- Splitting the work repeats activation preparation, launches or transfers, and the join waits for the slowest source, so it can be slower than one GPU operation.
What must stay true
- Each route position gives exactly one contribution, even when the same expert is chosen at two positions.
- Every source stays alive until every contribution has settled, even after another one has failed.
- The output is published only after every contribution has succeeded. A failure publishes nothing.
- Contributions are weighted once, before the join, and added in the declared position order.
crit-hub