crit-hub

Quality: perplexity and KL divergence

Guide What makes an engine's output drift from its checkpoint, how perplexity and KL divergence measure the drift, and how crit Eval turns it into a pass or a fail.

Same weights, different numbers

An inference engine runs a checkpoint, a model's published weights. Two engines that load the same checkpoint should give every next token the same probability, but fast engines change the arithmetic, so they differ a little. crit Eval calls the difference drift, and judges quality as exactly that: what the engine loses on top of the checkpoint the user chose, nothing else.

Where drift comes from:

Scoring a fixed text

The engine and a reference read the same fixed text. At each position both give a probability to every token in the vocabulary, and the next input is always the text's own token, never one they chose. This is teacher forcing: position 40 means the same thing on both sides.

The simplest score is the probability the model gave the token that actually comes next. Its negative logarithm is the negative log-likelihood (NLL), in nats: a token given 0.9 costs 0.11, one given 0.05 costs 3.0. Perplexity (PPL) is the exponential of the mean:

NLL = -ln p(actual next token)
PPL = exp(mean NLL over the text)

A perplexity of 3.4 means the model was, on average, as unsure as if it chose among 3.4 equally likely tokens.

The animation needs JavaScript; the text around it says the same in words.

crit Eval reports the engine's perplexity over the reference's on the same tokens, the perplexity ratio: 1.036 is 3.6% worse.

KL divergence compares whole distributions

Perplexity reads one number per position. KL divergence reads the whole next-token distribution: how far the engine's is from the reference's, with every token weighted by how likely the reference finds it.

KL(ref || engine) =
  sum over tokens t of
  p_ref(t) * ln(p_ref(t) / p_engine(t))

It is zero only when the two agree, never negative, and blind to which token the text holds. A token the reference gives 0.25 and the engine 0.10 adds 0.25 × ln 2.5 = 0.23 nats. A vocabulary often holds over 100,000 tokens, so crit Eval compares the tokens in both sides' 16 likeliest and lumps everything else into one bucket: a lower bound on the full KL, exact when the rest is negligible.

The animation needs JavaScript; the text around it says the same in words.

Why perplexity is not enough

Perplexity averages one token's NLL over hundreds of positions, and two kinds of change slip through. Errors cancel: less probability for the text's token here and more there leave the mean where it was. And probability can move among the alternatives without touching the text's token at all, while a sampler now writes different words.

An engine that makes the model's distributions sharper can even lower perplexity, since the text's token is often the likeliest one. crit Eval gives no credit for a ratio below 1.

The animation needs JavaScript; the text around it says the same in words.

The mean and the tail

A text gives one KL per position, and crit Eval keeps two summaries: the mean, and the 99th percentile (p99), the value 99% of positions stay at or below. The mean says how much the engine drifts overall. The p99 catches the few positions where the distribution moves far, which the mean dilutes: a digit or a bracket that flips at a few positions in a hundred barely moves the mean, and it is the error a reader notices.

In preserve engines, the tail tends to spend the budget first. One measured engine reports a mean KLD of 0.022 on its worst path, 44% of the 0.05 limit, and a p99 of 0.29, which sets its budget used: 57%.

The animation needs JavaScript; the text around it says the same in words.

How much drift: the quality modes

crit Eval's reference is the checkpoint itself, run in the model's own transformers code with FP32 activations and the stored weights dequantized exactly. It scores the quality texts once and stores each position's log-probability and top 16, so every engine of that checkpoint is judged against the same numbers.

A quality mode says how much drift from the checkpoint the user selected the engine may trade for speed. It is not about how that checkpoint is quantized or stored: a 16-bit, an FP8 or a 4-bit checkpoint takes either mode, and both judge against the same exact scores. crit Eval judges a run in one of the two, and crit-hub shows each result's:

For scale: on Qwen3.8-27B NVFP4, the checkpoint run with its activations quantized as it declares lands up to a mean KLD of 0.23 and a p99 of 3.7 from the exact scores, inside the speed budget. Results measured before crit-eval 0.5 judged speed with a budget twice preserve's, widened per set by the checkpoint's own drift up to today's limits. The KV cache stays exact in the reference in both modes, so quantizing it is drift either way. On crit-hub, the closed track takes either mode at that mode's own protocol budget, and results compare within one mode; an engine that runs 4-bit prompt activations is judged in speed.

The animation needs JavaScript; the text around it says the same in words.

From numbers to a pass or a fail

crit Eval's drift budget, from its protocol (quality.thresholds and quality.mode_thresholds):

Limit preserve speed
Perplexity ratio 1.05 1.4
Mean KLD 0.05 0.4
p99 KLD 0.5 4.0

The texts form four sets. Six conversations (prose, code, math, German, JSON, a story), each followed by a written answer cut to 256 tokens, are one. Three more put a long document before a conversation, so the prompt runs to about 1,024, 4,096 and 32,768 tokens, past the lengths where engines switch prefill kernels, prefetching or sparse attention.

Each set is judged on both paths the engine computes it with. SET/prompt is the prompt positions, from the prefill, judged by KLD alone: a chat model gives the user's own turn little probability, so its perplexity hangs on tokens the model calls near-impossible. SET/served is the answer (its first token from the prefill, the rest from the decode), judged on all three limits. A set's budget used is its largest share of any limit, the worst set's share is the quality figure, and every set must pass on both paths.

The figures are also reported by position (1-511, 512-1023, and so on), so a kernel switch at some length shows as a step. Past the 32K text, requests up to the full context the engine is built for must return facts placed at 2%, 50% and 97% of the prompt verbatim.

Three more rules:

The budget is there to be spent on speed: drifting less than allowed earns nothing.