crit-hub

fused-norms

Fused norms and rotary positions

Each norm, rotary rotation and gated product around a layer's matmuls runs as one kernel launch that reads its inputs once, each row's bits independent of the rows beside it.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Between its matmuls every layer normalizes, rotates and gates its activations: an RMSNorm with the residual add, query and key norms, rotary positions, the MLP's SiLU product.
  • Written in torch, each is a chain of small kernels, so a decode round launches hundreds of kernels that each move a few kilobytes, and a prompt pass reads the same activations several times.

How it works

  1. The model plugin calls each operation by what it computes, through the norms slot: rms_norm, rms_norm_rope, gated_rms_norm, silu_mul and the rest.
  2. Each call is one launch of the registry kernel crit-labs/fused-norms: it reads its inputs in their own dtype where they lie, computes in FP32 and rounds once.
  3. Where a layer runs two operations back to back, a fused form does both in one launch, such as the norm and rotation of a layer's queries and keys in rms_norm_rope_pair.
  4. A row's reduction runs in an order fixed by the row's width alone, so a token gives the same bits alone, in a verify window or in a prompt chunk.
  5. Without the kernel, the plugin runs torch-norms' arithmetic: the same operations as chains of torch kernels.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
flashinfertruetrue, falseFlashInfer (a provider) for the calls whose function it computes exactly: RMSNorm without a residual with a weight and output in the input dtype, and the SiLU and GELU products of packed BF16 or FP16 halves
configuration.json
{"plugins": {
  "use":     {"norms": "fused-norms"},
  "options": {"fused-norms": {"flashinfer": true}}
}}

Pays off when

  • Decode rounds, where an operation that took a chain of small kernels becomes one launch, and a captured round holds that many fewer nodes.
  • Long prompts, which read the same activations once instead of several times.

Doesn't pay off when

  • A matmul's epilogue already adds the residual or forms the gated product.
  • The layer uses a normalization the slot does not name, such as LayerNorm with a bias or a group norm.

What must stay true

  • Each operation computes in FP32 and rounds once, to the dtype asked for.
  • A row's bits never depend on the rows beside it or on how many rows share the call.
  • A fused form gives each of its outputs the bits the separate calls give it.