experts-reference
Reference experts block
The routed SwiGLU block of each token row over the experts' matrices, the reference every experts plugin must equal.
The problem
- A mixture-of-experts layer sends each token to a few experts, and every faster way to compute that (GPU kernels, a CPU tier, a split between the two) has to give the same result.
- That needs one block that is plainly right, however slow, to compare each of them with.
How it works
- The router has chosen each token's experts and their weights, in route order.
- Each chosen expert multiplies the token by its gate/up matrix, applies SwiGLU (the gate's SiLU times the up half), then multiplies by its down matrix.
- The experts' outputs are multiplied by their routing weights and summed, in route order.
- On the host it computes in FP64. Its device form runs the same block in torch, over the experts' matrices loaded on the device.
- It reads the weights from the
expert_sourceslot, so it needsexperts:source.
Configuration
No options: an engine selects it, and that is the whole setting.
configuration.json
{"plugins": {
"use": {"experts": "experts-reference"}
}}Pays off when
- An experts plugin is checked: its outputs are compared with this block's on the same weights.
- A model plugin of a mixture of experts is brought up, before any faster block is trusted.
Doesn't pay off when
- Serving at speed: a block computed token by token is far slower than a grouped kernel, or a CPU tier beside the GPU's expert cache.
What must stay true
- Each token's routes are taken in route order and summed in that order with their routing weights.
- It uses the expert source's weights as given, so a difference from it is the other plugin's.
crit-hub