crit-hub

host-expert-arithmetic

Bit-exact expert rows on the CPU

The CPU computes 4-bit NVFP4 expert rows in the exact operation order the GPU kernel declares, so an expert gives the same bits wherever it runs.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Where a missed expert runs changes with the cache state, the host's load and the request history, so two identical requests can split their experts differently.
  • Floating-point sums depend on their order. A CPU loop and a GPU kernel that group or fuse the same multiplications differently disagree in the last bits, and greedy output can fork.

How it works

  1. Each weight is a 4-bit E2M1 code, and each block of 16 has one 8-bit scale. Element k goes to accumulator lane k mod 16, so with the row laid out in blocks of 16, every column is a lane.
  2. Block by block, each lane rounds code times scale to FP32, then adds that times the activation with one fused multiply-add (FMA), which rounds once.
  3. The lanes are reduced by halving. Lane i adds lane i + 8, then i + 4, i + 2 and i + 1, rounding after every add, until lane 0 holds the sum.
  4. Lane 0 is multiplied by post, the global scale factor exactly as the GPU applies it.
  5. The CPU's AVX2 and AVX-512 vector code and 16 threads of a GPU warp do exactly these steps in this order, so they give the same bits. Any other order is a different answer.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
lanes161, 2, 4, 8, 16FP32 lanes of the GPU's reduction order the CPU repeats
reducehalvinghalving, sequentialHow the lanes are summed: pairwise halving or in sequence
productscaled_elementsscaled_elements, block_dotScale each element, then dot; or dot each block, then scale
activationsfp32fp32, e2m1x2, int8device form, nvfp4 matrices: what they multiply: FP32 activations (weight only, the 16-lane FP32 order), E2M1 x 2 per 16 under the matrix's input_scale (W4A4) or INT8 per 32 (W4A8), the integer block order; int4 matrices always W4A8 or W4A4
nativeautoauto, on, offdevice form: the native rows built at load (on: refuse to load without them), or the same orders in torch (off)
threads-1int, at least -1device form: threads a matrix's rows are split over (-1: physical cores)
configuration.json
{"plugins": {
  "use":     {"projections": "host-expert-arithmetic"},
  "options": {"host-expert-arithmetic": {"lanes": 16, "reduce": "halving", "product": "scaled_elements"}}
}}

Pays off when

  • Measured dispatch gives the CPU part of a step's NVFP4 misses, and the output must not depend on that split.
  • The GPU kernel's order can be declared, with one accumulator per lane, a fixed reduction tree and explicit rounding.

Doesn't pay off when

  • Every expert fits on the GPU, or measurement gives the CPU no share on this machine.
  • The GPU runs a library GEMM whose reduction order is not specified, so there is no order to reproduce.
  • The expert's activation uses CUDA's fast expf, which no host code reproduces. Then the expert still differs even though its rows match.

What must stay true

  • One FP32 rounding per declared operation, and no fused or reordered step the order does not name.
  • Lane assignment, block order and reduction pairing are identical on the CPU and the GPU.
  • Scales are decoded to the exact floats the GPU multiplies by, and post is applied the way the GPU applies it.
  • The activation and the sum over experts match too. The host returns one result per expert, and the GPU adds them in its plan order.