Learn
Learn explains how an inference engine turns a model's weights into answers, and the techniques that make it fast without changing what it says. Start with the guides. Every technique that fills a slot is a plugin, and its page shows the mechanism in an animation, its settings and what it must keep true.
Start here
How a crit engine is built
A crit engine is crit Core with plugins in its slots. Picture Core as a chain: some nodes are Core's own and the same in every engine, the others are slots, open until a plugin fills them.
How an inference engine works
What happens between a prompt and its reply, step by step, with crit Core as the worked example of how a real engine divides the work.
Quality: perplexity and KL divergence
What makes an engine's output drift from its checkpoint, how perplexity and KL divergence measure the drift, and how crit Eval turns it into a pass or a fail.
Glossary
The abbreviations and terms that come up on crit-hub, from tg and pp to PPL, KLD, GEMM and NVFP4, each in one line.
Architecture and placement plugins
Every tensor on the deviceEvery tensor goes on the device, with room kept free beside the plan; a model that does not fit is refused before anything loads.placementLayer offload for dense modelsA dense model too big for the GPU keeps some layers there and, for the rest, runs them on the CPU or streams their weights in, whichever plan measures faster.placement
Forward pass plugins
Embedding table in memoryThe embedding table is one tensor in memory, indexed by a step's token ids.embeddingsLookup-table rows from diskA large embedding or n-gram table stays on disk, and each token reads only its own rows, exactly and in parallel, instead of megabytes of read-ahead.embeddingsNorms in plain torchThe norms slot's plain plugin: every norm, rotary rotation and gated product as a chain of torch kernels, FP32 inside and one rounding, the reference fused-norms is tested against.normsFused norms and rotary positionsEach norm, rotary rotation and gated product around a layer's matmuls runs as one kernel launch that reads its inputs once, each row's bits independent of the rows beside it.normsDense projectionsRows times dense matrices, the plain projection every quantized plugin is tested against.projectionsBit-exact expert rows on the CPUThe CPU computes 4-bit NVFP4 expert rows in the exact operation order the GPU kernel declares, so an expert gives the same bits wherever it runs.projectionsLow-bit weight projectionsWeights are stored as low-bit codes with one scale per group of values, and the projection multiplies the codes directly, applying each scale once, so decoding reads far fewer bytes.projectionsSoftmax attentionExact causal softmax attention: each row's queries over its sequence's pages, read in place by the kernel that fits the page format.attentionPartitioned attentionA long sequence's keys and values are split across workers so none holds them all; each attends over its share, and the partials merge exactly by their softmax totals.attentionPaged KV cache with prefix sharingThe KV cache is cut into fixed pages, so requests that start with the same prompt point at the same pages instead of computing and storing them again.kv_cacheGated-delta recurrenceThe gated-delta recurrence token by token, keeping every state, with long prompt rows through a chunked kernel.recurrenceRecurrent state recoveryA recurrent layer mixes every token into one state, so after a speculative round the engine rebuilds that state for exactly the accepted tokens instead of dropping rejected rows.recurrenceTop-k routingEach token's top-k experts from one routing description: softmax or sigmoid scores, correction bias, group limits and renormalization.routerReference experts blockThe routed SwiGLU block of each token row over the experts' matrices, the reference every experts plugin must equal.expertsGrouped experts over bundlesA layer's routes are grouped by expert on the device and each expert's rows run once for all of its routes, in the format the checkpoint stores them, a route's bits independent of the rows beside it.expertsSplitting experts between CPU and GPUFor the experts the GPU does not hold, each step picks between copying them over and computing them on the CPU, choosing the split with the lowest measured time.expertsCPU expert tierA layer's share of routed experts runs on the CPU at FP32, with the device's integer rows and its plan order, so the sum is the one the GPU would give.expertsExpert weights in memoryExpert weights are held in memory as the checkpoint gives them. The plain source every other expert source is compared with.expert_sourceFile-backed experts with read-aheadExpert weights stay in a file on disk instead of memory: each expert is read as an exact, checked byte range, and the ones a predictor expects are read ahead.expert_sourceExpert cachingA mixture-of-experts model keeps its hot experts in GPU memory and fetches the rest from host memory on demand.expert_residencyExpert residencyAn expert cache that lets readers see only complete expert bundles, fills during decode but not prefill, and borrows memory the KV cache and prompt workspace are not using.expert_residency
Choosing tokens plugins
Structured output with speculative decodingA grammar matcher masks every token that would break the required format, and speculative rounds run it on a private copy that follows each draft and rewinds when drafts are rejected.samplerToken sampling on the GPUThe GPU applies the whole sampling law to the full vocabulary and checks sampled drafts by rejection sampling, so no logit row leaves the GPU and speculation keeps the output distribution.samplerMTP speculative decodingThe model's own multi-token prediction layer guesses the next few tokens, and one pass of the full model checks them all at once.drafterDraft treesThe drafter keeps its runner-up guesses as branches of a tree, one pass of the full model checks every branch, and the round keeps the longest path it agrees with.drafterverifierDraft vocabulary subsetThe drafter scores only the output-head rows of the tokens the model writes most, so drafts get cheaper, while verification keeps the full head and greedy output is unchanged.drafterHidden-state draftersA small published drafter reads hidden states from a few of the target model's layers and guesses the next tokens, one per step or a whole block per pass.drafterPrompt lookup decodingDrafts are copied from wherever the last few tokens appeared earlier in the prompt or output, and one pass of the full model checks them all.drafterChain verificationA round keeps the longest prefix of the draft chain the model agrees with, then the model's own token: the correction where a draft failed, a bonus at the end.verifier
Running the step plugins
Every step eagerEvery step runs as eager kernel launches: no capture, nothing to choose. The plain choice the graphs slot falls back to.graphsRecorded execution choicesFor each phase and shape, the engine picks eager launches, a full graph or a partial graph from measured costs, and records the pick so later processes repeat it.graphs
Inside Core and the model plugin
Techniques that fill no slot: Core does them for every engine, or a model plugin calls them.
Checked tensor loading
Each tensor is read from the checkpoint file at its exact byte range, checked against its digest, and copied with both ends kept alive until the copy is done.
Joining expert contributions
Expert outputs from the CPU and GPU arrive at different times; they are summed only once all have settled, in a fixed order, so arrival order never changes the result.
Batched serving
Requests share each step: all their prompt and decode rows go through one matrix multiply per weight, and each request's output stays exactly what it would be alone.
Prefix checkpoints across tiers
A prompt prefix's computed state is saved as one complete bundle on the GPU, in host memory or on disk, and restored instead of recomputed when that is faster.
Row compaction, metadata patches and retirement
One step serves several requests: only ready rows join, results return by request identity, metadata is patched instead of rebuilt, and memory is freed only once the GPU is done.
Speculative rounds on the GPU
The GPU drafts, verifies, accepts and moves on in two captured graphs, so the host waits once per speculative round instead of deciding between steps.
Host memory admission
Before a large host allocation, the engine checks what the operating system reports it can back (free memory, the commit limit, the container's cgroup), not only its own budget.
Host work mailbox
From inside a captured CUDA graph, the GPU posts work for the host into a shared mailbox, and the stream waits for the answer only where it reads it.
Checkpoint import audit
Before loading an imported checkpoint, the engine reads only its Safetensors headers and index, and checks that every tensor it needs is there with the exact type, shape and packed layout.
Chunked prefill with a latency target
A long prompt is read in chunks sized to keep each step under a latency target, through a cursor that moves on only when the model accepts a chunk.
crit-hub