experts
The routed experts' SwiGLU blocks for each token row, wherever the experts are.
optional
In crit's library
Reference experts blockThe routed SwiGLU block of each token row over the experts' matrices, the reference every experts plugin must equal.expertsGrouped experts over bundlesA layer's routes are grouped by expert on the device and each expert's rows run once for all of its routes, in the format the checkpoint stores them, a route's bits independent of the rows beside it.expertsSplitting experts between CPU and GPUFor the experts the GPU does not hold, each step picks between copying them over and computing them on the CPU, choosing the split with the lowest measured time.expertsCPU expert tierA layer's share of routed experts runs on the CPU at FP32, with the device's integer rows and its plan order, so the sum is the one the GPU would give.experts
Interface
The methods the engine calls on the slot's plugin, in their host form: token ids, rows of numbers and dicts. Every plugin of the slot has them, which is what lets an ablation switch one for another.
| Method | What it does |
|---|---|
run(...) | The routed block of each token row |
Device form: load(layer, expert, gate_up, down) and forward(layer, x, ids, weights)
crit-hub