crit-hub

Learn

Learn explains how an inference engine turns a model's weights into answers, and the techniques that make it fast without changing what it says. Start with the guides. Every technique that fills a slot is a plugin, and its page shows the mechanism in an animation, its settings and what it must keep true.

Start here

Architecture and placement plugins

Forward pass plugins

memory-table-rowsEmbedding table in memoryThe embedding table is one tensor in memory, indexed by a step's token ids.embeddingslookup-table-rowsLookup-table rows from diskA large embedding or n-gram table stays on disk, and each token reads only its own rows, exactly and in parallel, instead of megabytes of read-ahead.embeddingstorch-normsNorms in plain torchThe norms slot's plain plugin: every norm, rotary rotation and gated product as a chain of torch kernels, FP32 inside and one rounding, the reference fused-norms is tested against.normsfused-normsFused norms and rotary positionsEach norm, rotary rotation and gated product around a layer's matmuls runs as one kernel launch that reads its inputs once, each row's bits independent of the rows beside it.normsprojections-denseDense projectionsRows times dense matrices, the plain projection every quantized plugin is tested against.projectionshost-expert-arithmeticBit-exact expert rows on the CPUThe CPU computes 4-bit NVFP4 expert rows in the exact operation order the GPU kernel declares, so an expert gives the same bits wherever it runs.projectionsquantized-projectionsLow-bit weight projectionsWeights are stored as low-bit codes with one scale per group of values, and the projection multiplies the codes directly, applying each scale once, so decoding reads far fewer bytes.projectionssoftmax-attentionSoftmax attentionExact causal softmax attention: each row's queries over its sequence's pages, read in place by the kernel that fits the page format.attentionpartitioned-attentionPartitioned attentionA long sequence's keys and values are split across workers so none holds them all; each attends over its share, and the partials merge exactly by their softmax totals.attentionattention-statePaged KV cache with prefix sharingThe KV cache is cut into fixed pages, so requests that start with the same prompt point at the same pages instead of computing and storing them again.kv_cachesequential-recurrenceGated-delta recurrenceThe gated-delta recurrence token by token, keeping every state, with long prompt rows through a chunked kernel.recurrencerecurrent-state-recoveryRecurrent state recoveryA recurrent layer mixes every token into one state, so after a speculative round the engine rebuilds that state for exactly the accepted tokens instead of dropping rejected rows.recurrencetopk-routerTop-k routingEach token's top-k experts from one routing description: softmax or sigmoid scores, correction bias, group limits and renormalization.routerexperts-referenceReference experts blockThe routed SwiGLU block of each token row over the experts' matrices, the reference every experts plugin must equal.expertsgrouped-expertsGrouped experts over bundlesA layer's routes are grouped by expert on the device and each expert's rows run once for all of its routes, in the format the checkpoint stores them, a route's bits independent of the rows beside it.expertsheterogeneous-expert-dispatchSplitting experts between CPU and GPUFor the experts the GPU does not hold, each step picks between copying them over and computing them on the CPU, choosing the split with the lowest measured time.expertshost-expert-tierCPU expert tierA layer's share of routed experts runs on the CPU at FP32, with the device's integer rows and its plan order, so the sum is the one the GPU would give.expertsexpert-tensorsExpert weights in memoryExpert weights are held in memory as the checkpoint gives them. The plain source every other expert source is compared with.expert_sourcessd-expert-materializationFile-backed experts with read-aheadExpert weights stay in a file on disk instead of memory: each expert is read as an exact, checked byte range, and the ones a predictor expects are read ahead.expert_sourcedevice-expert-cacheExpert cachingA mixture-of-experts model keeps its hot experts in GPU memory and fetches the rest from host memory on demand.expert_residencyexpert-residencyExpert residencyAn expert cache that lets readers see only complete expert bundles, fills during decode but not prefill, and borrows memory the KV cache and prompt workspace are not using.expert_residency

Choosing tokens plugins

token-constraintsStructured output with speculative decodingA grammar matcher masks every token that would break the required format, and speculative rounds run it on a private copy that follows each draft and rewinds when drafts are rejected.samplertoken-samplingToken sampling on the GPUThe GPU applies the whole sampling law to the full vocabulary and checks sampled drafts by rejection sampling, so no logit row leaves the GPU and speculation keeps the output distribution.samplergreedy-head-drafterMTP speculative decodingThe model's own multi-token prediction layer guesses the next few tokens, and one pass of the full model checks them all at once.drafterdraft-treesDraft treesThe drafter keeps its runner-up guesses as branches of a tree, one pass of the full model checks every branch, and the round keeps the longest path it agrees with.drafterverifierdraft-vocabulary-subsetDraft vocabulary subsetThe drafter scores only the output-head rows of the tokens the model writes most, so drafts get cheaper, while verification keeps the full head and greedy output is unchanged.drafterhidden-state-draftersHidden-state draftersA small published drafter reads hidden states from a few of the target model's layers and guesses the next tokens, one per step or a whole block per pass.draftermtp-decodePrompt lookup decodingDrafts are copied from wherever the last few tokens appeared earlier in the prompt or output, and one pass of the full model checks them all.drafterchain-verifierChain verificationA round keeps the longest prefix of the draft chain the model agrees with, then the model's own token: the correction where a draft failed, a bonus at the end.verifier

Running the step plugins

Inside Core and the model plugin

Techniques that fill no slot: Core does them for every engine, or a model plugin calls them.

Checked tensor loading Each tensor is read from the checkpoint file at its exact byte range, checked against its digest, and copied with both ends kept alive until the copy is done. Joining expert contributions Expert outputs from the CPU and GPU arrive at different times; they are summed only once all have settled, in a fixed order, so arrival order never changes the result. Batched serving Requests share each step: all their prompt and decode rows go through one matrix multiply per weight, and each request's output stays exactly what it would be alone. Prefix checkpoints across tiers A prompt prefix's computed state is saved as one complete bundle on the GPU, in host memory or on disk, and restored instead of recomputed when that is faster. Row compaction, metadata patches and retirement One step serves several requests: only ready rows join, results return by request identity, metadata is patched instead of rebuilt, and memory is freed only once the GPU is done. Speculative rounds on the GPU The GPU drafts, verifies, accepts and moves on in two captured graphs, so the host waits once per speculative round instead of deciding between steps. Host memory admission Before a large host allocation, the engine checks what the operating system reports it can back (free memory, the commit limit, the container's cgroup), not only its own budget. Host work mailbox From inside a captured CUDA graph, the GPU posts work for the host into a shared mailbox, and the stream waits for the answer only where it reads it. Checkpoint import audit Before loading an imported checkpoint, the engine reads only its Safetensors headers and index, and checks that every tensor it needs is there with the exact type, shape and packed layout. Chunked prefill with a latency target A long prompt is read in chunks sized to keep each step under a latency target, through a cursor that moves on only when the model accepts a chunk.