crit-hub

grouped-experts

Grouped experts over bundles

A layer's routes are grouped by expert on the device and each expert's rows run once for all of its routes, in the format the checkpoint stores them, a route's bits independent of the rows beside it.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A mixture-of-experts layer sends each token to a few experts. Run route by route, every route reads its expert's matrices again.
  • Batching routes must not change a route's result: a token decoded alone, in a batch or in a prompt chunk has to give the same bits, or batched requests decode other text.

How it works

  1. Each expert is one contiguous bundle in its stored format (bf16, fp8, nvfp4, int4 or mxfp4), held by the plugin or by a residency cache's slot store and found through a slot table.
  2. A layer's routes are sorted by expert on the device, stably, and cut into items of 8 routes for decode or 32 for prompt chunks, with no host synchronization.
  3. The registry kernel crit-labs/grouped-experts decodes each weight exactly to BF16 and runs every expert's gate/up and down rows once for all of its routes, with crit's declared SwiGLU between them.
  4. Each token's routes are summed in route order, so the sum is the same whichever tier computed a route.
  5. Without the kernel it runs the same operands in torch, in chunks of one fixed size, so a route runs the same shapes in any call.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
activationsbf16bf16, nvfp4what NVFP4 experts multiply: bf16, 16-bit activations everywhere (W4A16, weight only); nvfp4, each NVFP4 layer as its stored matrices declare: activations and intermediates quantized to NVFP4 at each matrix's input_scale where they carry one (W4A4), 16-bit activations where they do not (W4A16); the facts name each layer's
arithmetictensortensor, declaredtensor: BF16 tensor cores for every format (the fastest); declared: crit's integer block order (host-expert-arithmetic's host_int.h), the bits crit's CPU expert tier gives on the same bundles, for NVFP4 experts W4A4 (with activations nvfp4) and int4 experts W4A8 (8-bit activations, a quantization of its own for speed runs), so a dispatch that sends some routes to the CPU changes no token; only the registry kernel computes it (refused without it)
workspace_mib512int 16 to 65536the route rows a call keeps at once: a longer prompt chunk runs in parts of the tokens that fit (whole multiples of 256), each part combined into its tokens' rows as it goes, which changes no route's bits; a prompt-sized call of fewer tokens runs in a kept prompt workspace's buffers
kerneltruetrue, falseuse the registry kernel crit-labs/grouped-experts where it loads; false runs the torch arithmetic
configuration.json
{"plugins": {
  "use":     {"experts": "grouped-experts"},
  "options": {"grouped-experts": {"activations": "bf16", "arithmetic": "tensor", "workspace_mib": 512}}
}}

Pays off when

  • A model with routed SwiGLU experts whose matrix sizes are multiples of 64, in one of the formats it reads.
  • Decode rounds and prompt chunks alike, since one kernel serves both and a call at a fixed token count is captured in a CUDA graph.

Doesn't pay off when

  • Experts that are not SwiGLU, or matrix sizes that are not multiples of 64.
  • A provider's grouped GEMM that is faster on the GPU can replace it only if it keeps a route's bits independent of the rows beside it.

What must stay true

  • A route's result is one function of the route and its expert: the same alone or batched, in any chunk, decode or prompt.
  • Each token's routes are summed in route order.
  • Weights are decoded exactly as stored: nothing is re-encoded.