crit-hub

norms

The elementwise work between a layer's matmuls: the RMSNorm before each block with its residual add, query and key norms, rotary positions, gated norms and the activation products of the MLP, each row computed on its own so its bits never depend on the rows beside it.

optional

In crit's library

Interface

The methods the engine calls on the slot's plugin, in their host form: token ids, rows of numbers and dicts. Every plugin of the slot has them, which is what lets an ablation switch one for another.

MethodWhat it does
rms_norm_rows(rows, weight, eps, ...)Each row normalized by its root mean square and scaled by its weight, after a residual add when one is given
gated_rms_norm_rows(rows, gates, weight, eps, ...)The norm times an activation of the gate, element by element
rope_rows(rows, positions, freqs, ...)Each head rotated at its token's position, in pairs on its first rotary values
act_mul_rows(...)An activation of the gate times the up half, as an MLP computes it
l2_norm_rows(rows, eps)Each row divided by its length
stream_sum_rows(...), stream_add_rows(...)Several residual streams mixed into one row, or one row added into every stream
softplus_decay_rows(...)A recurrent layer's log-decay from a projection's output

Device form: rms_norm, rms_norm_heads, rms_norm_rope, gated_rms_norm, rope, silu_mul, gelu_mul, sigmoid_mul, l2_norm, stream_sum, stream_add and softplus_decay over a step's tensors, each row's bits independent of the rows beside it