Same weights, different numbers
An inference engine runs a checkpoint, a model's published weights. Two engines that load the same checkpoint should give every next token the same probability, but fast engines change the arithmetic, so they differ a little. crit Eval calls the difference drift, and judges quality as exactly that: what the engine loses on top of the checkpoint the user chose, nothing else.
Where drift comes from:
- Quantized weights and activations. A checkpoint may store its weights in 8 bits (FP8) or 4 bits (NVFP4: 4-bit values with one scale per group of 16). An engine that also rounds the activations, the numbers fed into each matrix multiply, to 8 or 4 bits runs faster and moves the output. See Quantized projections.
- KV cache encodings. The keys and values every later token reads (Paged KV cache) can be stored in 8 bits to fit a longer context. In one engine, an FP8 cache used 114% of the quality budget on 32K-token prompts; an INT8 cache with a scale per position fit.
- Approximate or reordered arithmetic. FP16 accumulation, fused kernels and another summation order each round differently. The differences are tiny, but where a model's expert routing has near-ties, one can change which experts a token uses, and the change carries through the rest of the text.
- Sampling bugs. A sampler that ignores top-p, applies a server's default penalty or accepts speculative drafts by greedy match draws from the wrong distribution even when every probability is right. crit Eval checks sampling separately (Token sampling); this guide is about the probabilities.
Scoring a fixed text
The engine and a reference read the same fixed text. At each position both give a probability to every token in the vocabulary, and the next input is always the text's own token, never one they chose. This is teacher forcing: position 40 means the same thing on both sides.
The simplest score is the probability the model gave the token that actually comes next. Its negative logarithm is the negative log-likelihood (NLL), in nats: a token given 0.9 costs 0.11, one given 0.05 costs 3.0. Perplexity (PPL) is the exponential of the mean:
NLL = -ln p(actual next token)
PPL = exp(mean NLL over the text)
A perplexity of 3.4 means the model was, on average, as unsure as if it chose among 3.4 equally likely tokens.
crit Eval reports the engine's perplexity over the reference's on the same tokens, the perplexity ratio: 1.036 is 3.6% worse.
KL divergence compares whole distributions
Perplexity reads one number per position. KL divergence reads the whole next-token distribution: how far the engine's is from the reference's, with every token weighted by how likely the reference finds it.
KL(ref || engine) =
sum over tokens t of
p_ref(t) * ln(p_ref(t) / p_engine(t))
It is zero only when the two agree, never negative, and blind to which token the text holds. A token the reference gives 0.25 and the engine 0.10 adds 0.25 × ln 2.5 = 0.23 nats. A vocabulary often holds over 100,000 tokens, so crit Eval compares the tokens in both sides' 16 likeliest and lumps everything else into one bucket: a lower bound on the full KL, exact when the rest is negligible.
Why perplexity is not enough
Perplexity averages one token's NLL over hundreds of positions, and two kinds of change slip through. Errors cancel: less probability for the text's token here and more there leave the mean where it was. And probability can move among the alternatives without touching the text's token at all, while a sampler now writes different words.
An engine that makes the model's distributions sharper can even lower perplexity, since the text's token is often the likeliest one. crit Eval gives no credit for a ratio below 1.
The mean and the tail
A text gives one KL per position, and crit Eval keeps two summaries: the mean, and the 99th percentile (p99), the value 99% of positions stay at or below. The mean says how much the engine drifts overall. The p99 catches the few positions where the distribution moves far, which the mean dilutes: a digit or a bracket that flips at a few positions in a hundred barely moves the mean, and it is the error a reader notices.
In preserve engines, the tail tends to spend the budget first. One measured engine reports a mean KLD of 0.022 on its worst path, 44% of the 0.05 limit, and a p99 of 0.29, which sets its budget used: 57%.
How much drift: the quality modes
crit Eval's reference is the checkpoint itself, run in the model's own transformers code with FP32 activations and the stored weights dequantized exactly. It scores the quality texts once and stores each position's log-probability and top 16, so every engine of that checkpoint is judged against the same numbers.
A quality mode says how much drift from the checkpoint the user selected the engine may trade for speed. It is not about how that checkpoint is quantized or stored: a 16-bit, an FP8 or a 4-bit checkpoint takes either mode, and both judge against the same exact scores. crit Eval judges a run in one of the two, and crit-hub shows each result's:
preservekeeps the output close to the checkpoint's. Every precision change the engine makes counts as drift: on Qwen3.8-27B NVFP4, 4-bit prompt activations used 252% of its budget.speed, crit Eval's default, trades drift for speed within a budget eight times preserve's. It keeps out a broken engine (skipped layers, dropped experts, a collapsed path) rather than rationing precision, so 4-bit and 8-bit activations, a lower-precision KV cache or head fit it. Two replies to the same request may also differ where they part at a near-tie, which is all that rounding on two processors (an expert on the CPU in one request, on the GPU in the next) can cause.
For scale: on Qwen3.8-27B NVFP4, the checkpoint run with its activations quantized as it declares lands up to a mean KLD of 0.23 and a p99 of 3.7 from the exact scores, inside the speed budget. Results measured before crit-eval 0.5 judged speed with a budget twice preserve's, widened per set by the checkpoint's own drift up to today's limits. The KV cache stays exact in the reference in both modes, so quantizing it is drift either way. On crit-hub, the closed track takes either mode at that mode's own protocol budget, and results compare within one mode; an engine that runs 4-bit prompt activations is judged in speed.
From numbers to a pass or a fail
crit Eval's drift budget, from its protocol (quality.thresholds and quality.mode_thresholds):
| Limit | preserve |
speed |
|---|---|---|
| Perplexity ratio | 1.05 | 1.4 |
| Mean KLD | 0.05 | 0.4 |
| p99 KLD | 0.5 | 4.0 |
The texts form four sets. Six conversations (prose, code, math, German, JSON, a story), each followed by a written answer cut to 256 tokens, are one. Three more put a long document before a conversation, so the prompt runs to about 1,024, 4,096 and 32,768 tokens, past the lengths where engines switch prefill kernels, prefetching or sparse attention.
Each set is judged on both paths the engine computes it with. SET/prompt is the prompt positions, from the prefill, judged by KLD alone: a chat model gives the user's own turn little probability, so its perplexity hangs on tokens the model calls near-impossible. SET/served is the answer (its first token from the prefill, the rest from the decode), judged on all three limits. A set's budget used is its largest share of any limit, the worst set's share is the quality figure, and every set must pass on both paths.
The figures are also reported by position (1-511, 512-1023, and so on), so a kernel switch at some length shows as a step. Past the 32K text, requests up to the full context the engine is built for must return facts placed at 2%, 50% and 97% of the prompt verbatim.
Three more rules:
- Scores from the serving path. The log-probabilities of the engine's greedy chat replies must match its scores of the same tokens within 0.001 nats.
- The reference's own spread. Some models cannot match themselves: two valid FP32 runs of Qwen3.8-Flash-Next's 32K text differ by a mean of 0.29 nats per position near its end. The reference runs once more with TF32 inputs, and a set's limits rise to twice that spread where it is larger.
- Fail fast. A set that drifts a mean KLD over ten times the budget's stops the scoring of the rest.
The budget is there to be spent on speed: drifting less than allowed earns nothing.
crit-hub