crit-hub

torch-norms

Norms in plain torch

The norms slot's plain plugin: every norm, rotary rotation and gated product as a chain of torch kernels, FP32 inside and one rounding, the reference fused-norms is tested against.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Every layer normalizes, rotates and gates its activations between its matmuls, and a faster form, such as one fused kernel, must give exactly the bits the plain arithmetic gives.
  • That needs one plugin that computes each operation plainly, on any device torch runs on, to compare the others with and to fall back to.

How it works

  1. On the host it computes each operation in FP64 from the inputs as given: the slot's oracle.
  2. On the device each operation is a chain of torch kernels (a cast, a square, a mean, an rsqrt, the multiplies, a cast back), FP32 inside and one rounding at the end.
  3. Every reduction goes through torch's fused rms_norm, which reduces one row at a time, so a row's bits do not depend on the rows beside it.
  4. Its calls are capture-safe, so a captured decode round can run them.

Configuration

No options: an engine selects it, and that is the whole setting.

configuration.json
{"plugins": {
  "use": {"norms": "torch-norms"}
}}

Pays off when

  • A model plugin is brought up, before a fused kernel is trusted.
  • A norms plugin is checked: fused-norms is compared with it, operation by operation.

Doesn't pay off when

  • Decode rounds, where each operation launches several small kernels that each move a few kilobytes. fused-norms runs each as one launch.

What must stay true

  • FP32 inside and one rounding, to the dtype asked for.
  • A row's bits never depend on the rows beside it or on how many rows share the call.