fused-norms
Fused norms and rotary positions
Each norm, rotary rotation and gated product around a layer's matmuls runs as one kernel launch that reads its inputs once, each row's bits independent of the rows beside it.
The problem
- Between its matmuls every layer normalizes, rotates and gates its activations: an RMSNorm with the residual add, query and key norms, rotary positions, the MLP's SiLU product.
- Written in torch, each is a chain of small kernels, so a decode round launches hundreds of kernels that each move a few kilobytes, and a prompt pass reads the same activations several times.
How it works
- The model plugin calls each operation by what it computes, through the norms slot:
rms_norm,rms_norm_rope,gated_rms_norm,silu_muland the rest. - Each call is one launch of the registry kernel
crit-labs/fused-norms: it reads its inputs in their own dtype where they lie, computes in FP32 and rounds once. - Where a layer runs two operations back to back, a fused form does both in one launch, such as the norm and rotation of a layer's queries and keys in
rms_norm_rope_pair. - A row's reduction runs in an order fixed by the row's width alone, so a token gives the same bits alone, in a verify window or in a prompt chunk.
- Without the kernel, the plugin runs torch-norms' arithmetic: the same operations as chains of torch kernels.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
flashinfer | true | true, false | FlashInfer (a provider) for the calls whose function it computes exactly: RMSNorm without a residual with a weight and output in the input dtype, and the SiLU and GELU products of packed BF16 or FP16 halves |
configuration.json
{"plugins": {
"use": {"norms": "fused-norms"},
"options": {"fused-norms": {"flashinfer": true}}
}}Pays off when
- Decode rounds, where an operation that took a chain of small kernels becomes one launch, and a captured round holds that many fewer nodes.
- Long prompts, which read the same activations once instead of several times.
Doesn't pay off when
- A matmul's epilogue already adds the residual or forms the gated product.
- The layer uses a normalization the slot does not name, such as LayerNorm with a bias or a group norm.
What must stay true
- Each operation computes in FP32 and rounds once, to the dtype asked for.
- A row's bits never depend on the rows beside it or on how many rows share the call.
- A fused form gives each of its outputs the bits the separate calls give it.
crit-hub