quantized-projections
Low-bit weight projections
Weights are stored as low-bit codes with one scale per group of values, and the projection multiplies the codes directly, applying each scale once, so decoding reads far fewer bytes.
The problem
- Decoding reads every weight of every layer for each token, so memory traffic, not arithmetic, sets the pace.
- FP32 or BF16 weights spend 2 to 4 bytes per value, and many models do not fit the GPU at all in that form.
How it works
- The row is cut into groups of 32 values, and each group gets one scale, its largest magnitude divided by the largest code (7 for 4 bits, 127 for 8).
- Each weight becomes the nearest whole number of steps, a signed code; two 4-bit codes share a byte, the even element in the low nibble.
- The projection multiplies codes by the activations and sums each group, then multiplies that sum by the group's scale once.
- Dynamic A8 also encodes the activations per group of 32 as they arrive, so the inner sum is integer by integer and takes both scales.
- FP8 rows (one E4M3 byte per weight, one BF16 scale per row) and NVFP4 (E2M1 codes, an E4M3 scale per 16 values, one FP32 divisor per tensor) follow the same pattern with other code tables.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
format | int8 | int8, int4, nvfp4 | The code the weights are stored in: int8 or int4 with one FP32 scale per group, or nvfp4 |
group | 32 | int 1 to 4096 | weights per FP32 scale, for int8 and int4 |
four_over_six | false | true, false | for nvfp4: scale blocks to 4 instead of 6 where that lowers the error |
activations | nvfp4 | nvfp4, bf16 | on the device, what a stored NVFP4 matrix multiplies: NVFP4 activations (W4A4: prompt rows through cuBLASLt, decode rows as decode_activations says) or 16-bit ones (weight only: GEMVs for decode rows, the matrix decoded for the prompt GEMMs) |
fp8_activations | fp8 | fp8, bf16 | FP8 matrices' prompt rows: fp8 (W8A8: each activation row in E4M3 by its own scale, cuBLASLt with the row and column scales in its epilogue), or bf16 (W8A16: the E4M3 weights widened exactly to BF16 times the 16-bit activations, FP32 sums, the row scale after; on the tile up to tile_rows, past it the matrix a chunk of rows at a time for a BF16 GEMM): an exact variant a preserve-mode engine may take where activation quantization costs too much quality. Decode rows with decode_activations prompt follow it. The facts give every format's arithmetic (prompt_arithmetic) |
decode_activations | prompt | prompt, bf16 | on the device, what decode rows (calls of up to the facts' row_invariance rows: decode steps, verify windows) of FP8 and W4A4 NVFP4 matrices multiply: prompt, the prompt rows' activations (each row quantized to E4M3, or to NVFP4 at the input scale), so a decoded token and the same token in a prompt (a replay, a prompt cache) differ only in their sums' order; bf16, the 16-bit activations themselves (weight only: W8A16, W4A16), closer to the unquantized model and further from the prompt rows. Weight-only NVFP4 (activations bf16) and MXFP8 matrices multiply 16-bit activations in both phases. With prompt, W4A4 NVFP4 decode rows run mma-rows' block-scaled FP4 MMA where the device has it (the sm_120 family; E2M1 codes with their block scales on both operands, FP32 sums, the prompt rows' arithmetic in another order), else the tile on the quantized values as BF16 |
decode_formats | "" | str | comma-separated format=activations pairs (formats fp8 and nvfp4, activations prompt or bf16) that override decode_activations for the matrices of that format, chosen per matrix at load from its format, e.g. nvfp4=prompt,fp8=bf16: W4A4 NVFP4 decode rows on the prompt rows' NVFP4 activations (mma-rows' FP4 MMA where the device has it) and FP8 decode rows on 16-bit activations (W8A16: decode-gemv's dot GEMV with gemv auto); a format it does not name takes decode_activations. The facts' decode_activations (per format) and decode_kernels (per role) say which each matrix took |
decode_rows | 16 | int 8 to 16 | the most rows a decode call takes (8 to 16), each row's bits those of the row alone (the facts' row_invariance rows; 8 on the Triton GEMV): the CUDA-core GEMVs grow compute bound toward 16 rows on wide matrices (a vocabulary-wide FP8 head is about 2.6x cuBLASLt there), so a model that runs 9-16 decode rows mostly in such matrices may take 8 and send the rest as prompt rows |
tile_rows | auto | auto, 0, 64, 128, 192, 256 | on mma-rows' tile (gemv mma), the most prompt rows a call sends through the tile too, 64 rows a kernel call: auto, by each matrix's prompt arithmetic as far as the tile beats its prompt path (weight-only prompts 64; W4A4, W8A8 and BF16 none: measured, device_matrices.AUTO_TILES), or one count for every matrix. A row gets the same bits in any call of up to that many rows where the decode and prompt arithmetic agree (the facts' row_invariance.rows: the fewest over the loaded matrices), so captured prompt buckets that size equal the unpadded chunk. Not for weight-only NVFP4 prompts in fp16_acc or fp8, nor with prompt_gemm fixed_tiles |
prompt | bf16 | bf16, fp16_acc, fp8 | with activations bf16, the prompt GEMMs' arithmetic: the matrix decoded to BF16 (FP32 accumulation), to FP16 accumulating in FP16 (twice the rate, FP16 out), or to FP8 rows against FP8 activation rows (rowwise scaled) |
prompt_gemm | provider | provider, fixed_tiles, pinned, tile | the GEMM prompt rows (calls of more rows than a decode call takes) go through: provider, torch's (cuBLASLt picks its algorithm, and so a row's sums, by the row count); fixed_tiles, for BF16 matrices and weight-only ones with prompt bf16, one whose tiles never change an output's order (crit-labs/fixed-tile-gemm; one row at a time through torch where it is not here, correct and much slower); pinned, for every format (W4A4, W8A8, W8A16, W4A16 with prompt bf16, MXFP8, BF16), cuBLASLt with one split-K-free algorithm per GEMM shape and format, chosen at load and run for every row count (crit-labs/pinned-gemm; one row a call through the provider where it is not here, correct and much slower); tile, for every format with gemv mma (W4A16 with prompt bf16), crit-labs/mma-rows' prompt tile: each prompt row multiplies the decode rows' activations in the decode tile's order, so every output has the bits the same row gets as a decode row (the facts' row_invariance prompt_exact: a scored token's log-probability is the served one's, preserve mode's exact honesty) at GEMM-like weight reuse; a matrix the decode tile does not take, or a kernel build without the prompt tile, goes through the provider and the facts say prompt_exact false. With fixed_tiles, pinned or tile a prompt row's result is the same at every prompt length, in any chunk, padded bucket or pass shared with other sequences (the facts' row_invariance prompt_rows), and prompt rows skip mma-rows' tile (tile_rows applies to decode rows only) |
prompt_roles | "" | str | with activations bf16, comma-separated role=arithmetic pairs (bf16, fp16_acc, fp8) that override prompt for the matrices a model loads with that role (load(tensors, format, role)), e.g. gate_up=fp8 for FP8 rows on the widest matrix and prompt elsewhere; a role it does not name takes prompt; prompt_gemm fixed_tiles applies to the bf16 ones |
gemv | auto | auto, mma, native, triton | the decode rows' kernel: auto, each matrix's fastest measured one: FP8 matrices' 16-bit decode rows (W8A16) on crit-labs/decode-gemv's dot GEMV (a Triton program of 16 rows by a block of the matrix on tensor cores, a fixed split of K added in order; faster than the tile from 8 rows on, level below; K a multiple of 128), W4A4 decode rows on crit-labs/mma-rows' block-scaled FP4 MMA (the sm_120 family; K a multiple of 128), every other matrix as mma (the facts' decode_kernels name each role's); mma, crit-labs/mma-rows' tensor-core tile (every format, the weights read once a call whatever the rows, the prompt rows' quantized activations; K a multiple of 32), else as native; native, crit-labs/decode-gemv's CUDA-core GEMVs (its Triton GEMV for NVFP4 matrices whose width is no multiple of 32); triton, its Triton GEMV for NVFP4 (BF16 rows then run a row at a time through torch); where no kernel is here, the torch fallbacks (the matrix's exact values, one row at a time). Each keeps FP32 sums and a row's bits alone and batched; their orders differ, so a process uses one a matrix |
configuration.json
{"plugins": {
"use": {"projections": "quantized-projections"},
"options": {"quantized-projections": {"format": "int8", "group": 32, "four_over_six": false}}
}}Pays off when
- Decoding is bound by reading weights, and fewer bytes per weight cut that traffic.
- A model has to fit in GPU memory, or as many of its weights as possible do.
Doesn't pay off when
- The rounding error matters for the task, so measure quality on the model at hand; 4-bit codes and quantized activations change the arithmetic.
- The checkpoint uses another layout, scale precision or packing; it needs its own decoder or an explicit conversion, never a relabel.
- Decoding the codes costs more than the bytes it saves for the workload; a path that first expands the whole matrix back to floats saves nothing.
What must stay true
- Codes, packing order and scale indexing agree exactly; the group of element i is i / 32 and its scale is stored in the precision the format declares.
- A format is identified by its declared layout, never inferred from byte counts.
- Any change of reduction order, activation encoding or fused rounding is a new arithmetic and is checked on its own.
crit-hub