host-expert-arithmetic
Bit-exact expert rows on the CPU
The CPU computes 4-bit NVFP4 expert rows in the exact operation order the GPU kernel declares, so an expert gives the same bits wherever it runs.
The problem
- Where a missed expert runs changes with the cache state, the host's load and the request history, so two identical requests can split their experts differently.
- Floating-point sums depend on their order. A CPU loop and a GPU kernel that group or fuse the same multiplications differently disagree in the last bits, and greedy output can fork.
How it works
- Each weight is a 4-bit E2M1 code, and each block of 16 has one 8-bit scale. Element k goes to accumulator lane k mod 16, so with the row laid out in blocks of 16, every column is a lane.
- Block by block, each lane rounds code times scale to FP32, then adds that times the activation with one fused multiply-add (FMA), which rounds once.
- The lanes are reduced by halving. Lane i adds lane i + 8, then i + 4, i + 2 and i + 1, rounding after every add, until lane 0 holds the sum.
- Lane 0 is multiplied by post, the global scale factor exactly as the GPU applies it.
- The CPU's AVX2 and AVX-512 vector code and 16 threads of a GPU warp do exactly these steps in this order, so they give the same bits. Any other order is a different answer.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
lanes | 16 | 1, 2, 4, 8, 16 | FP32 lanes of the GPU's reduction order the CPU repeats |
reduce | halving | halving, sequential | How the lanes are summed: pairwise halving or in sequence |
product | scaled_elements | scaled_elements, block_dot | Scale each element, then dot; or dot each block, then scale |
activations | fp32 | fp32, e2m1x2, int8 | device form, nvfp4 matrices: what they multiply: FP32 activations (weight only, the 16-lane FP32 order), E2M1 x 2 per 16 under the matrix's input_scale (W4A4) or INT8 per 32 (W4A8), the integer block order; int4 matrices always W4A8 or W4A4 |
native | auto | auto, on, off | device form: the native rows built at load (on: refuse to load without them), or the same orders in torch (off) |
threads | -1 | int, at least -1 | device form: threads a matrix's rows are split over (-1: physical cores) |
configuration.json
{"plugins": {
"use": {"projections": "host-expert-arithmetic"},
"options": {"host-expert-arithmetic": {"lanes": 16, "reduce": "halving", "product": "scaled_elements"}}
}}Pays off when
- Measured dispatch gives the CPU part of a step's NVFP4 misses, and the output must not depend on that split.
- The GPU kernel's order can be declared, with one accumulator per lane, a fixed reduction tree and explicit rounding.
Doesn't pay off when
- Every expert fits on the GPU, or measurement gives the CPU no share on this machine.
- The GPU runs a library GEMM whose reduction order is not specified, so there is no order to reproduce.
- The expert's activation uses CUDA's fast
expf, which no host code reproduces. Then the expert still differs even though its rows match.
What must stay true
- One FP32 rounding per declared operation, and no fused or reordered step the order does not name.
- Lane assignment, block order and reduction pairing are identical on the CPU and the GPU.
- Scales are decoded to the exact floats the GPU multiplies by, and post is applied the way the GPU applies it.
- The activation and the sum over experts match too. The host returns one result per expert, and the GPU adds them in its plan order.
crit-hub