torch-norms
Norms in plain torch
The norms slot's plain plugin: every norm, rotary rotation and gated product as a chain of torch kernels, FP32 inside and one rounding, the reference fused-norms is tested against.
The problem
- Every layer normalizes, rotates and gates its activations between its matmuls, and a faster form, such as one fused kernel, must give exactly the bits the plain arithmetic gives.
- That needs one plugin that computes each operation plainly, on any device torch runs on, to compare the others with and to fall back to.
How it works
- On the host it computes each operation in FP64 from the inputs as given: the slot's oracle.
- On the device each operation is a chain of torch kernels (a cast, a square, a mean, an rsqrt, the multiplies, a cast back), FP32 inside and one rounding at the end.
- Every reduction goes through torch's fused
rms_norm, which reduces one row at a time, so a row's bits do not depend on the rows beside it. - Its calls are capture-safe, so a captured decode round can run them.
Configuration
No options: an engine selects it, and that is the whole setting.
configuration.json
{"plugins": {
"use": {"norms": "torch-norms"}
}}Pays off when
- A model plugin is brought up, before a fused kernel is trusted.
- A norms plugin is checked:
fused-normsis compared with it, operation by operation.
Doesn't pay off when
- Decode rounds, where each operation launches several small kernels that each move a few kilobytes.
fused-normsruns each as one launch.
What must stay true
- FP32 inside and one rounding, to the dtype asked for.
- A row's bits never depend on the rows beside it or on how many rows share the call.
crit-hub