projections-dense
Dense projections
Rows times dense matrices, the plain projection every quantized plugin is tested against.
The problem
- Most of a model's arithmetic is projections: each layer multiplies its rows by weight matrices, y = W x.
- Low-bit formats (int8, int4, NVFP4) make those matrices smaller and faster to read, and each is compared with the plain product to show what it costs in accuracy.
How it works
- On the host it multiplies each row by the matrix in FP64 with exact sums, one dot product per output.
- On the device it keeps the matrix in BF16 and multiplies with torch.
- It takes 16-bit matrices only: weights stored as NVFP4 or FP8 are refused (it conflicts with
stored:nvfp4andstored:fp8).
Configuration
No options: an engine selects it, and that is the whole setting.
configuration.json
{"plugins": {
"use": {"projections": "projections-dense"}
}}Pays off when
- The weights are BF16 and fit on the device.
- A low-bit format is measured:
quantized-projectionsis compared with it.
Doesn't pay off when
- Decode waits on reading weights, so a low-bit format that reads a quarter of the bytes is faster.
- The checkpoint stores its weights as NVFP4 or FP8. A plugin that reads them as stored is needed.
What must stay true
- The host form's sums are exact, so a difference from it is the other plugin's rounding.
- On the device, each row's result has the bits of that row computed alone, in a call of up to
DECODE_ROWSrows.
crit-hub