grouped-experts
Grouped experts over bundles
A layer's routes are grouped by expert on the device and each expert's rows run once for all of its routes, in the format the checkpoint stores them, a route's bits independent of the rows beside it.
The problem
- A mixture-of-experts layer sends each token to a few experts. Run route by route, every route reads its expert's matrices again.
- Batching routes must not change a route's result: a token decoded alone, in a batch or in a prompt chunk has to give the same bits, or batched requests decode other text.
How it works
- Each expert is one contiguous bundle in its stored format (bf16, fp8, nvfp4, int4 or mxfp4), held by the plugin or by a residency cache's slot store and found through a slot table.
- A layer's routes are sorted by expert on the device, stably, and cut into items of 8 routes for decode or 32 for prompt chunks, with no host synchronization.
- The registry kernel
crit-labs/grouped-expertsdecodes each weight exactly to BF16 and runs every expert's gate/up and down rows once for all of its routes, with crit's declared SwiGLU between them. - Each token's routes are summed in route order, so the sum is the same whichever tier computed a route.
- Without the kernel it runs the same operands in torch, in chunks of one fixed size, so a route runs the same shapes in any call.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
activations | bf16 | bf16, nvfp4 | what NVFP4 experts multiply: bf16, 16-bit activations everywhere (W4A16, weight only); nvfp4, each NVFP4 layer as its stored matrices declare: activations and intermediates quantized to NVFP4 at each matrix's input_scale where they carry one (W4A4), 16-bit activations where they do not (W4A16); the facts name each layer's |
arithmetic | tensor | tensor, declared | tensor: BF16 tensor cores for every format (the fastest); declared: crit's integer block order (host-expert-arithmetic's host_int.h), the bits crit's CPU expert tier gives on the same bundles, for NVFP4 experts W4A4 (with activations nvfp4) and int4 experts W4A8 (8-bit activations, a quantization of its own for speed runs), so a dispatch that sends some routes to the CPU changes no token; only the registry kernel computes it (refused without it) |
workspace_mib | 512 | int 16 to 65536 | the route rows a call keeps at once: a longer prompt chunk runs in parts of the tokens that fit (whole multiples of 256), each part combined into its tokens' rows as it goes, which changes no route's bits; a prompt-sized call of fewer tokens runs in a kept prompt workspace's buffers |
kernel | true | true, false | use the registry kernel crit-labs/grouped-experts where it loads; false runs the torch arithmetic |
configuration.json
{"plugins": {
"use": {"experts": "grouped-experts"},
"options": {"grouped-experts": {"activations": "bf16", "arithmetic": "tensor", "workspace_mib": 512}}
}}Pays off when
- A model with routed SwiGLU experts whose matrix sizes are multiples of 64, in one of the formats it reads.
- Decode rounds and prompt chunks alike, since one kernel serves both and a call at a fixed token count is captured in a CUDA graph.
Doesn't pay off when
- Experts that are not SwiGLU, or matrix sizes that are not multiples of 64.
- A provider's grouped GEMM that is faster on the GPU can replace it only if it keeps a route's bits independent of the rows beside it.
What must stay true
- A route's result is one function of the route and its expert: the same alone or batched, in any chunk, decode or prompt.
- Each token's routes are summed in route order.
- Weights are decoded exactly as stored: nothing is re-encoded.
crit-hub