crit-hub

Glossary

Guide The abbreviations and terms that come up on crit-hub, from tg and pp to PPL, KLD, GEMM and NVFP4, each in one line.

Inference has a lot of short names. Each one here gets a line that says what it stands for and what it means, plus a link to the page that explains it when there is one.

Speed

Term Means
pp prompt processing Reading the prompt, also called prefill. All prompt tokens go through the model in one pass, so it is limited by arithmetic. pp512 means a 512-token prompt.
tg token generation Writing the reply, also called decode: one pass of the model per new token, limited by how fast the weights can be read. tg128 means 128 generated tokens.
prefill, decode The two phases above. How an inference engine works shows both.
tok/s tokens per second Throughput. Decode tok/s is how fast a reply streams, and prefill tok/s is how fast a prompt is read.
TTFT time to first token The wait from sending a request to seeing the first token of the reply: mostly prefill.
batch Several requests that share each pass of the model. See Batched serving.
speculative decoding A cheap drafter guesses several tokens and one pass of the full model checks them all. See MTP speculative decoding.

Quality

Term Means
NLL negative log-likelihood How surprised the model was by the actual next token: -ln p(token). Lower is better.
PPL perplexity exp(mean NLL) over a text. A PPL of 3.4 means the model was as unsure as if it chose among 3.4 equally likely tokens.
KL, KLD Kullback-Leibler divergence How far the engine's next-token distribution is from a reference's, at one position. 0 means identical.
mean KLD, p99 KLD The average KLD over every scored token, and the 99th percentile: the value only 1 token in 100 exceeds.
p99 99th percentile The value 99% of the samples stay under; it shows the worst cases that an average hides.
drift How far an engine's output moves from its checkpoint. Quality: perplexity and KL divergence explains how crit Eval measures it.

Arithmetic

Term Means
GEMM general matrix-matrix multiply A weight matrix times many rows at once. Prefill and batched decode are mostly GEMMs.
GEMV general matrix-vector multiply A weight matrix times one row. Decode of a single request is mostly GEMVs, and their speed is set by memory bandwidth.
MLP multi-layer perceptron The feed-forward block of each layer, usually its largest weights.
FMA fused multiply-add a × b + c in one instruction with one rounding.
FLOP/s floating-point operations per second Arithmetic speed. TFLOP/s is 10^12 per second.
logits The model's raw score for every token in its vocabulary, before sampling turns them into probabilities. See Token sampling on the GPU.

Number formats

Term Means
FP32 32-bit float Full precision, the usual reference.
BF16 brain float 16 16 bits with FP32's range (8 exponent bits) and less precision (7 mantissa bits). The common format for published weights.
FP16 16-bit float 16 bits with more precision than BF16 and a much smaller range.
FP8 8-bit float Usually E4M3 for weights and activations, with a scale per tensor, row or channel.
E4M3, E2M1 exponent and mantissa bits How a small float splits its bits: E4M3 has 4 exponent and 3 mantissa bits, and E2M1 has 2 and 1.
NVFP4 NVIDIA 4-bit float E2M1 values with one E4M3 scale per 16 values and one FP32 scale per tensor. See Low-bit weight projections.
INT8, INT4 8- and 4-bit integers Integer codes with a scale per group, row or token.
W4A16, W8A8 weight and activation bits W4A16 means 4-bit weights with 16-bit activations; W8A8 means both are 8-bit.
quantization Storing weights, activations or the KV cache in fewer bits to read fewer bytes, at some cost in drift.

Model parts

Term Means
KV cache key-value cache Every past token's attention keys and values, kept so each new token reads them instead of recomputing them. See Paged KV cache.
GQA, MQA grouped-query, multi-query attention Several query heads share one key and value head, so the KV cache is smaller.
RoPE rotary position embedding Encodes a token's position by rotating its queries and keys.
MoE mixture of experts A layer with many expert MLPs, of which a router picks a few per token. See Expert caching.
MTP multi-token prediction An extra layer trained with the model to predict tokens further ahead, used as a drafter.
EAGLE, DFlash Published drafters: EAGLE reads the target model's hidden states, DFlash drafts a block per pass. See Hidden-state drafters.
gated delta A recurrent layer that keeps a fixed-size state instead of a growing KV cache. See Recurrent state recovery.

Hardware

Term Means
VRAM video RAM The GPU's own memory, where the weights and the KV cache need to fit.
PCIe PCI Express The bus between the CPU's memory and the GPU, much slower than VRAM.
CUDA NVIDIA's platform for programs on its GPUs.
CUDA graph A recorded sequence of GPU work replayed with one call, which saves the cost of launching each kernel.
kernel One function that runs on the GPU, such as a GEMV or an attention step.
sm_90, sm_120 streaming multiprocessor version A GPU generation's feature level: sm_90 is Hopper, sm_120 is consumer Blackwell.
AVX2, AVX-512 advanced vector extensions x86 CPU instructions that work on 8 or 16 numbers at once.
SSD solid-state drive Fast disk storage, used to hold weights that do not fit in memory. See File-backed experts.