Inference has a lot of short names. Each one here gets a line that says what it stands for and what it means, plus a link to the page that explains it when there is one.
Speed
| Term | Means |
|---|---|
| pp prompt processing | Reading the prompt, also called prefill. All prompt tokens go through the model in one pass, so it is limited by arithmetic. pp512 means a 512-token prompt. |
| tg token generation | Writing the reply, also called decode: one pass of the model per new token, limited by how fast the weights can be read. tg128 means 128 generated tokens. |
| prefill, decode | The two phases above. How an inference engine works shows both. |
| tok/s tokens per second | Throughput. Decode tok/s is how fast a reply streams, and prefill tok/s is how fast a prompt is read. |
| TTFT time to first token | The wait from sending a request to seeing the first token of the reply: mostly prefill. |
| batch | Several requests that share each pass of the model. See Batched serving. |
| speculative decoding | A cheap drafter guesses several tokens and one pass of the full model checks them all. See MTP speculative decoding. |
Quality
| Term | Means |
|---|---|
| NLL negative log-likelihood | How surprised the model was by the actual next token: -ln p(token). Lower is better. |
| PPL perplexity | exp(mean NLL) over a text. A PPL of 3.4 means the model was as unsure as if it chose among 3.4 equally likely tokens. |
| KL, KLD Kullback-Leibler divergence | How far the engine's next-token distribution is from a reference's, at one position. 0 means identical. |
| mean KLD, p99 KLD | The average KLD over every scored token, and the 99th percentile: the value only 1 token in 100 exceeds. |
| p99 99th percentile | The value 99% of the samples stay under; it shows the worst cases that an average hides. |
| drift | How far an engine's output moves from its checkpoint. Quality: perplexity and KL divergence explains how crit Eval measures it. |
Arithmetic
| Term | Means |
|---|---|
| GEMM general matrix-matrix multiply | A weight matrix times many rows at once. Prefill and batched decode are mostly GEMMs. |
| GEMV general matrix-vector multiply | A weight matrix times one row. Decode of a single request is mostly GEMVs, and their speed is set by memory bandwidth. |
| MLP multi-layer perceptron | The feed-forward block of each layer, usually its largest weights. |
| FMA fused multiply-add | a × b + c in one instruction with one rounding. |
| FLOP/s floating-point operations per second | Arithmetic speed. TFLOP/s is 10^12 per second. |
| logits | The model's raw score for every token in its vocabulary, before sampling turns them into probabilities. See Token sampling on the GPU. |
Number formats
| Term | Means |
|---|---|
| FP32 32-bit float | Full precision, the usual reference. |
| BF16 brain float 16 | 16 bits with FP32's range (8 exponent bits) and less precision (7 mantissa bits). The common format for published weights. |
| FP16 16-bit float | 16 bits with more precision than BF16 and a much smaller range. |
| FP8 8-bit float | Usually E4M3 for weights and activations, with a scale per tensor, row or channel. |
| E4M3, E2M1 exponent and mantissa bits | How a small float splits its bits: E4M3 has 4 exponent and 3 mantissa bits, and E2M1 has 2 and 1. |
| NVFP4 NVIDIA 4-bit float | E2M1 values with one E4M3 scale per 16 values and one FP32 scale per tensor. See Low-bit weight projections. |
| INT8, INT4 8- and 4-bit integers | Integer codes with a scale per group, row or token. |
| W4A16, W8A8 weight and activation bits | W4A16 means 4-bit weights with 16-bit activations; W8A8 means both are 8-bit. |
| quantization | Storing weights, activations or the KV cache in fewer bits to read fewer bytes, at some cost in drift. |
Model parts
| Term | Means |
|---|---|
| KV cache key-value cache | Every past token's attention keys and values, kept so each new token reads them instead of recomputing them. See Paged KV cache. |
| GQA, MQA grouped-query, multi-query attention | Several query heads share one key and value head, so the KV cache is smaller. |
| RoPE rotary position embedding | Encodes a token's position by rotating its queries and keys. |
| MoE mixture of experts | A layer with many expert MLPs, of which a router picks a few per token. See Expert caching. |
| MTP multi-token prediction | An extra layer trained with the model to predict tokens further ahead, used as a drafter. |
| EAGLE, DFlash | Published drafters: EAGLE reads the target model's hidden states, DFlash drafts a block per pass. See Hidden-state drafters. |
| gated delta | A recurrent layer that keeps a fixed-size state instead of a growing KV cache. See Recurrent state recovery. |
Hardware
| Term | Means |
|---|---|
| VRAM video RAM | The GPU's own memory, where the weights and the KV cache need to fit. |
| PCIe PCI Express | The bus between the CPU's memory and the GPU, much slower than VRAM. |
| CUDA | NVIDIA's platform for programs on its GPUs. |
| CUDA graph | A recorded sequence of GPU work replayed with one call, which saves the cost of launching each kernel. |
| kernel | One function that runs on the GPU, such as a GEMV or an attention step. |
| sm_90, sm_120 streaming multiprocessor version | A GPU generation's feature level: sm_90 is Hopper, sm_120 is consumer Blackwell. |
| AVX2, AVX-512 advanced vector extensions | x86 CPU instructions that work on 8 or 16 numbers at once. |
| SSD solid-state drive | Fast disk storage, used to hold weights that do not fit in memory. See File-backed experts. |
crit-hub