crit-hub

topk-router

Top-k routing

Each token's top-k experts from one routing description: softmax or sigmoid scores, correction bias, group limits and renormalization.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A mixture-of-experts layer sends each token to a few of its experts, chosen from the router's logits. Models choose differently: softmax or sigmoid scores, a correction bias, groups of experts, renormalized weights.
  • The GPU kernel and every host check must apply exactly the model's rule, or a token reaches other experts.

How it works

  1. The router's logits become scores by a softmax over the experts or a sigmoid each, before or after the selection (score_before_selection).
  2. A per-expert correction bias, where the model has one, changes which experts are picked, never their weights.
  3. With groups, the experts are split into groups, each group is scored by its best group_score_top experts, and only the best selected_groups groups take part.
  4. The top k experts are chosen, ties going to the lower expert id. Their weights are renormalized to sum to one if renormalize is set, then multiplied by scaling.
  5. Its device form runs the same rule over a step's router logits in torch.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
scoringsoftmaxsoftmax, sigmoidHow router logits become weights: softmax over the experts, or a sigmoid each
score_before_selectionfalsetrue, falseScore every expert before the top-k choice, not only the chosen ones
renormalizefalsetrue, falseRescale the chosen experts' weights to sum to one
scaling1float, at least 0Factor every routed weight is multiplied by
groups1int, at least 1Expert groups for group-limited routing (1: no groups)
selected_groups1int, at least 1Groups a token may route into
group_score_top1int, at least 1Top experts per group whose scores rank the group
kerneltruetrue, falseselect through the registry kernel crit-labs/topk-routing where it loads (rules without a group limit); false: torch
configuration.json
{"plugins": {
  "use":     {"router": "topk-router"},
  "options": {"topk-router": {"scoring": "softmax", "score_before_selection": false, "renormalize": false}}
}}

Pays off when

  • The model routes its experts by a rule its options describe.
  • A device routing kernel is checked: it is compared with this rule on the model's settings.

Doesn't pay off when

  • The model routes by a rule its options cannot describe, which then needs a router of its own.

What must stay true

  • Every token gets exactly k experts, and a tie goes to the lower expert id.
  • The bias changes only which experts are picked, never their weights.
  • The device form gives the host form's experts and weights.