norms
The elementwise work between a layer's matmuls: the RMSNorm before each block with its residual add, query and key norms, rotary positions, gated norms and the activation products of the MLP, each row computed on its own so its bits never depend on the rows beside it.
optional
In crit's library
Norms in plain torchThe norms slot's plain plugin: every norm, rotary rotation and gated product as a chain of torch kernels, FP32 inside and one rounding, the reference fused-norms is tested against.normsFused norms and rotary positionsEach norm, rotary rotation and gated product around a layer's matmuls runs as one kernel launch that reads its inputs once, each row's bits independent of the rows beside it.norms
Interface
The methods the engine calls on the slot's plugin, in their host form: token ids, rows of numbers and dicts. Every plugin of the slot has them, which is what lets an ablation switch one for another.
| Method | What it does |
|---|---|
rms_norm_rows(rows, weight, eps, ...) | Each row normalized by its root mean square and scaled by its weight, after a residual add when one is given |
gated_rms_norm_rows(rows, gates, weight, eps, ...) | The norm times an activation of the gate, element by element |
rope_rows(rows, positions, freqs, ...) | Each head rotated at its token's position, in pairs on its first rotary values |
act_mul_rows(...) | An activation of the gate times the up half, as an MLP computes it |
l2_norm_rows(rows, eps) | Each row divided by its length |
stream_sum_rows(...), stream_add_rows(...) | Several residual streams mixed into one row, or one row added into every stream |
softplus_decay_rows(...) | A recurrent layer's log-decay from a projection's output |
Device form: rms_norm, rms_norm_heads, rms_norm_rope, gated_rms_norm, rope, silu_mul, gelu_mul, sigmoid_mul, l2_norm, stream_sum, stream_add and softplus_decay over a step's tensors, each row's bits independent of the rows beside it
crit-hub