crit-hub

How a crit engine is built

Guide A crit engine is crit Core with plugins in its slots. Picture Core as a chain: some nodes are Core's own and the same in every engine, the others are slots, open until a plugin fills them.

The animation needs JavaScript; the text around it says the same in words.

crit Core: a chain with open slots

A request runs down the chain from left to right. The solid nodes are Core's own: it loads the engine, takes the request in, schedules many requests into one step, constrains structured output and streams the reply back. The open nodes are slots, one for each part that depends on the model or the GPU.

NodeWhat happens here
loadReads the engine's configuration.json, checks that its plugins fit together and puts each in its slot
placementrequiredWhere each tensor lives: GPU, host memory or disk
requestA request comes in: the chat template, the prompt's tokens, admission
scheduleRows of many requests go into one step
graphsWhich captured CUDA graph a step runs
modelrequiredThe model's architecture: its layers, run over the slots after it
embeddingsToken ids to vectors
normsNormalization, rotary positions and gated products around the matmuls
projectionsrequiredEvery matrix multiply
attentionrequiredEach token looks back over the ones before it
kv_cacherequiredKeys and values kept for every request
recurrenceThe state of linear-attention layers, in hybrid models
routerPicks each token's experts, in mixture-of-experts models
expertsRuns the picked experts
expert_sourceWhere expert weights come from
expert_residencyWhich experts stay on the GPU
constrainStructured output: the tokens a row may take next
samplerrequiredPicks the next token
drafterGuesses the next few tokens ahead
verifierKeeps the guesses the model agrees with
streamStop rules, the reply streamed back, crit serve and crit-eval

Six slots every engine fills: model, placement, projections, attention, kv_cache and sampler. The others only when it needs them: a model without experts leaves the four expert slots open, and an engine without drafts leaves drafter and verifier open.

An engine fills the slots

An engine names one plugin for each slot it uses, in its configuration.json. Here is an example: crit Core with crit's library plugins in its slots and a model plugin of its own, for a dense model that drafts nothing.

configuration.json
{
  "plugins": {"use": {
    "placement": "device-placement",
    "graphs": "recorded-selection",
    "model": "my-model",
    "embeddings": "memory-table-rows",
    "norms": "torch-norms",
    "projections": "quantized-projections",
    "attention": "softmax-attention",
    "kv_cache": "attention-state",
    "sampler": "token-sampling"
  }}
}

Where plugins come from

How Core checks an engine