crit-hub

How an inference engine works

Guide What happens between a prompt and its reply, step by step, with crit Core as the worked example of how a real engine divides the work.

An inference engine runs a trained model for requests: it turns the prompt into tokens, runs the model, picks the next token and repeats until the reply is done, for many requests at once. Around that one calculation sits a lot of plumbing. crit Core is that plumbing as a library, and the engine built on it supplies the model. Core's own names appear in code, so each idea maps onto crit's source.

Text becomes tokens, and tokens become text

A model reads and writes tokens: integer ids from a fixed vocabulary, each standing for a piece of text such as a word, part of a word or a single byte. A tokenizer turns the prompt into ids, after a chat template has rendered the conversation as one text. A detokenizer turns the model's ids back into text.

Detokenizing happens one token at a time, and a token can end inside a character: é is two bytes in UTF-8, and a byte-level tokenizer may write them as two tokens. So the decoder holds an incomplete character back until the token that completes it, and never takes released text back. Stop strings need the same care. A client can ask the reply to end at ###, and the stop can span tokens, so the stop matcher holds back only text that could still begin a stop.

The animation needs JavaScript; the text around it says the same in words.

In crit the tokenizer belongs to the engine, which hands Core a TextFrontend. Core supplies the streaming decoder (offset_decoder) and the stop matching (StopMatcher) on top. Stop tokens are simpler: the model's end-of-text ids end the request as soon as one is written.

A request's life

To the engine, a request is a prompt, an output budget (the most tokens it may write) and a few options. Core runs its life in five parts:

  1. Admission. The request waits in a queue until it fits. Core keeps the memory budgets but never guesses what a request costs: the model says whether it fits, and reserves its memory through Core's ledger when its session opens.
  2. Prefill. The engine computes the whole prompt, in chunks Core grants (prefill_chunk, 1,024 tokens by default).
  3. Decode. Then one new token per step. The engine launches each step on the GPU and returns at once with a readiness handle, which Core waits on.
  4. Commit and publication. A new token is provisional. Core checks it against stop tokens, stop strings and the output budget, calls commit() with how many it accepts, and only then publishes it.
  5. Retirement. When the request ends (a stop, its budget, a cancellation or a failure), Core waits until all the device work the request launched has settled, then closes its session and returns its memory.

The animation needs JavaScript; the text around it says the same in words.

Keeping commit apart from computing pays off later: a speculative step computes several tokens and may keep only some, and a stop string can cut a step short. Cancellation takes the same road as any other ending: no more work, then retirement once in-flight work settles. At shutdown or after a failure Core drains the engine's backend, settling every outstanding piece of device work, so memory is never freed while a kernel may still read it.

What one step computes

Each step is a forward pass. The model's layers turn the input tokens into one vector per position, and a final projection turns the last one into logits: one score per vocabulary entry. Most of the pass is matrix multiplication against the weights, which for a large model are tens of gigabytes.

Prefill sends a whole chunk of prompt tokens through each weight matrix together, so every byte read from memory serves many tokens, and a long prompt is limited by arithmetic. Decode sends one token through and still reads every weight, so its speed is set by memory bandwidth. For example, 16 GB of weights at 1.6 TB/s take at least 10 ms per pass: at most 100 tokens per second for one request, however fast the arithmetic.

The animation needs JavaScript; the text around it says the same in words.

Most of what fast engines do follows from this. They read fewer bytes per pass (quantized projections), get several tokens from one pass (speculative decoding), or serve several requests in each pass (below).

The KV cache and other state

Attention lets each new token look back at every earlier one: it compares itself with each earlier token's key and reads a mix of their values. To avoid recomputing those at every step, each attention layer keeps every token's key and value: the KV cache, one row per token. Hybrid models add recurrent layers, which keep a fixed-size state per request instead.

Core keeps the books for this state without touching its bytes. Its StateCache is a ledger of KV pages (fixed blocks of rows) and recurrent state slots. It decides which ids hold what; the engine allocates the memory, runs the kernels that write it and makes the copies. One decode step uses it like this:

Round r = cache.begin(seq, 1);       // the new row: page, offset
// the engine's kernels write K and V there
cache.commit(seq, r.id, accepted);   // keep accepted rows
cache.retire(cache.epoch());         // after readiness

Two rules keep it safe. A freed page is handed out again only after retire(), called once the device has finished reading it. And a new prompt may share another request's full, committed pages when its tokens match exactly, as Paged KV cache shows.

Picking the next token

Logits are scores, not a choice. Greedy decoding takes the highest. Sampling turns the scores into weights and draws: temperature sharpens or flattens them, top_k keeps the k highest, top_p keeps the smallest set holding a share p of the weight, and a random number between 0 and 1 picks among what is left.

The animation needs JavaScript; the text around it says the same in words.

Core's Sampler applies these rules, with logit bias and repetition penalties, in one fixed order. Its random number is a pure function of the request's seed and the token's position, so a redone step picks the same token with no generator state to rewind. The runtime itself never samples: the session calls the sampler after reading logits, or follows the same rules on the GPU. A token constraint plugs in here, masking every token that would break a JSON schema or grammar before the draw.

Many requests at once

A decode pass reads every weight anyway, so serving several requests in one pass costs little more than serving one. That is batching, the source of a serving engine's throughput. Its limit is memory, since every active request holds its own KV cache.

Core schedules this with a bounded number of lanes: max_concurrency active requests (at most 4,096), each admitted by the model. Admission is first come, first served by default. With a lookahead, a request that fits may pass one waiting for memory, but by default a waiting request can be passed only 8 times, so nobody waits forever.

With mixed batches, one call into the engine carries prefill and decode rows together under a per-step token budget (max_batch_tokens). Each row first gets its minimum, one token for a decoder, and the rest goes to prompt chunks, so decoders keep moving while a long prompt is read. Core publishes nothing from a group until the whole group is ready.

The animation needs JavaScript; the text around it says the same in words.

What Core owns and what the engine owns

Core owns The engine owns
The worker, the queue and admission The model's layers, kernels and logits
Resource budgets and reservations Weight, tensor and state layouts
Readiness, commit and publication Device work: launches, streams, graphs
Cancellation, retirement and draining Tokenization (TextFrontend)

The engine plugs in through three interfaces. A Model declares what it supports in its ModelInfo (vocabulary, context, stop tokens, concurrency, sampling options) and opens a Session per request. A Session holds one request's state and answers advance() to compute within a budget, commit() to keep what Core accepted and close() once its work has settled. A Backend stands for the device; its drain() returns only when all submitted work has finished. An engine serving many requests implements ConcurrentModel instead: the same contract over groups of rows, plus admission() to say whether a request fits and retire() to certify that a finished request's work has settled.

Everything else stays inside the engine: dense or mixture-of-experts, CPU or GPU, caches, graph capture. Core's libraries for text, sampling, state, model files and constraints are there to use, not required. The Core + Inference walkthrough builds the smallest complete engine: a four-token model whose session runs one quantized projection, takes the highest logit and leaves the rest to Core.