topk-router
Top-k routing
Each token's top-k experts from one routing description: softmax or sigmoid scores, correction bias, group limits and renormalization.
The problem
- A mixture-of-experts layer sends each token to a few of its experts, chosen from the router's logits. Models choose differently: softmax or sigmoid scores, a correction bias, groups of experts, renormalized weights.
- The GPU kernel and every host check must apply exactly the model's rule, or a token reaches other experts.
How it works
- The router's logits become scores by a softmax over the experts or a sigmoid each, before or after the selection (
score_before_selection). - A per-expert correction bias, where the model has one, changes which experts are picked, never their weights.
- With groups, the experts are split into
groups, each group is scored by its bestgroup_score_topexperts, and only the bestselected_groupsgroups take part. - The top k experts are chosen, ties going to the lower expert id. Their weights are renormalized to sum to one if
renormalizeis set, then multiplied byscaling. - Its device form runs the same rule over a step's router logits in torch.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
scoring | softmax | softmax, sigmoid | How router logits become weights: softmax over the experts, or a sigmoid each |
score_before_selection | false | true, false | Score every expert before the top-k choice, not only the chosen ones |
renormalize | false | true, false | Rescale the chosen experts' weights to sum to one |
scaling | 1 | float, at least 0 | Factor every routed weight is multiplied by |
groups | 1 | int, at least 1 | Expert groups for group-limited routing (1: no groups) |
selected_groups | 1 | int, at least 1 | Groups a token may route into |
group_score_top | 1 | int, at least 1 | Top experts per group whose scores rank the group |
kernel | true | true, false | select through the registry kernel crit-labs/topk-routing where it loads (rules without a group limit); false: torch |
configuration.json
{"plugins": {
"use": {"router": "topk-router"},
"options": {"topk-router": {"scoring": "softmax", "score_before_selection": false, "renormalize": false}}
}}Pays off when
- The model routes its experts by a rule its options describe.
- A device routing kernel is checked: it is compared with this rule on the model's settings.
Doesn't pay off when
- The model routes by a rule its options cannot describe, which then needs a router of its own.
What must stay true
- Every token gets exactly k experts, and a tie goes to the lower expert id.
- The bias changes only which experts are picked, never their weights.
- The device form gives the host form's experts and weights.
crit-hub