crit-hub

experts-reference

Reference experts block

The routed SwiGLU block of each token row over the experts' matrices, the reference every experts plugin must equal.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A mixture-of-experts layer sends each token to a few experts, and every faster way to compute that (GPU kernels, a CPU tier, a split between the two) has to give the same result.
  • That needs one block that is plainly right, however slow, to compare each of them with.

How it works

  1. The router has chosen each token's experts and their weights, in route order.
  2. Each chosen expert multiplies the token by its gate/up matrix, applies SwiGLU (the gate's SiLU times the up half), then multiplies by its down matrix.
  3. The experts' outputs are multiplied by their routing weights and summed, in route order.
  4. On the host it computes in FP64. Its device form runs the same block in torch, over the experts' matrices loaded on the device.
  5. It reads the weights from the expert_source slot, so it needs experts:source.

Configuration

No options: an engine selects it, and that is the whole setting.

configuration.json
{"plugins": {
  "use": {"experts": "experts-reference"}
}}

Pays off when

  • An experts plugin is checked: its outputs are compared with this block's on the same weights.
  • A model plugin of a mixture of experts is brought up, before any faster block is trusted.

Doesn't pay off when

  • Serving at speed: a block computed token by token is far slower than a grouped kernel, or a CPU tier beside the GPU's expert cache.

What must stay true

  • Each token's routes are taken in route order and summed in that order with their routing weights.
  • It uses the expert source's weights as given, so a difference from it is the other plugin's.