crit-hub

Joining expert contributions

Experts Expert outputs from the CPU and GPU arrive at different times; they are summed only once all have settled, in a fixed order, so arrival order never changes the result.

In an engine

No slot of its own: it lends its code to heterogeneous-expert-dispatch.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • When some of a token's experts run on the CPU and others on the GPU, their outputs arrive at different times, in a different order from run to run.
  • Floating-point addition rounds after every step, so a sum taken in arrival order can change in its last digits from one run to the next.
  • Releasing an expert's weights, or publishing the output, while another expert is still running would let that expert read freed memory or expose a half-finished result.

How it works

  1. The token's routing has one position per chosen expert, and each position owes exactly one contribution, already multiplied by its routing weight.
  2. Each position's work is launched on the CPU or the GPU and handed to the join with a receipt, which reports when the work is done, and its source, the weights it reads. The join keeps every source alive.
  3. Contributions finish in any order. The join is ready only when every position has been handed in and every receipt reports done.
  4. The join then adds the contributions in position order, never in arrival order. Only now are the sources released and the sum published.
  5. If a contribution failed, the join still waits for all the others to finish, then raises the error and publishes nothing.

Pays off when

  • One routed operation launches independent work on several sources, such as CPU and GPU experts, and must produce one result.
  • Runs must give the same output whatever the timing, so the sum cannot depend on which source was faster.

Doesn't pay off when

  • One fused operation already produces the whole result, so there is nothing to join.
  • Splitting the work repeats activation preparation, launches or transfers, and the join waits for the slowest source, so it can be slower than one GPU operation.

What must stay true

  • Each route position gives exactly one contribution, even when the same expert is chosen at two positions.
  • Every source stays alive until every contribution has settled, even after another one has failed.
  • The output is published only after every contribution has succeeded. A failure publishes nothing.
  • Contributions are weighted once, before the join, and added in the declared position order.