crit-hub

projections-dense

Dense projections

Rows times dense matrices, the plain projection every quantized plugin is tested against.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Most of a model's arithmetic is projections: each layer multiplies its rows by weight matrices, y = W x.
  • Low-bit formats (int8, int4, NVFP4) make those matrices smaller and faster to read, and each is compared with the plain product to show what it costs in accuracy.

How it works

  1. On the host it multiplies each row by the matrix in FP64 with exact sums, one dot product per output.
  2. On the device it keeps the matrix in BF16 and multiplies with torch.
  3. It takes 16-bit matrices only: weights stored as NVFP4 or FP8 are refused (it conflicts with stored:nvfp4 and stored:fp8).

Configuration

No options: an engine selects it, and that is the whole setting.

configuration.json
{"plugins": {
  "use": {"projections": "projections-dense"}
}}

Pays off when

  • The weights are BF16 and fit on the device.
  • A low-bit format is measured: quantized-projections is compared with it.

Doesn't pay off when

  • Decode waits on reading weights, so a low-bit format that reads a quarter of the bytes is faster.
  • The checkpoint stores its weights as NVFP4 or FP8. A plugin that reads them as stored is needed.

What must stay true

  • The host form's sums are exact, so a difference from it is the other plugin's rounding.
  • On the device, each row's result has the bits of that row computed alone, in a call of up to DECODE_ROWS rows.