crit Core: a chain with open slots
A request runs down the chain from left to right. The solid nodes are Core's own: it loads the engine, takes the request in, schedules many requests into one step, constrains structured output and streams the reply back. The open nodes are slots, one for each part that depends on the model or the GPU.
- loadcrit Core
- placementslot, required
- requestcrit Core
- schedulecrit Core
- graphsslot
- modelslot, required
- embeddingsslot
- normsslot
- projectionsslot, required
- attentionslot, required
- kv_cacheslot, required
- recurrenceslot
- routerslot
- expertsslot
- expert_sourceslot
- expert_residencyslot
- constraincrit Core
- samplerslot, required
- drafterslot
- verifierslot
- streamcrit Core
| Node | What happens here |
|---|---|
| load | Reads the engine's configuration.json, checks that its plugins fit together and puts each in its slot |
| placementrequired | Where each tensor lives: GPU, host memory or disk |
| request | A request comes in: the chat template, the prompt's tokens, admission |
| schedule | Rows of many requests go into one step |
| graphs | Which captured CUDA graph a step runs |
| modelrequired | The model's architecture: its layers, run over the slots after it |
| embeddings | Token ids to vectors |
| norms | Normalization, rotary positions and gated products around the matmuls |
| projectionsrequired | Every matrix multiply |
| attentionrequired | Each token looks back over the ones before it |
| kv_cacherequired | Keys and values kept for every request |
| recurrence | The state of linear-attention layers, in hybrid models |
| router | Picks each token's experts, in mixture-of-experts models |
| experts | Runs the picked experts |
| expert_source | Where expert weights come from |
| expert_residency | Which experts stay on the GPU |
| constrain | Structured output: the tokens a row may take next |
| samplerrequired | Picks the next token |
| drafter | Guesses the next few tokens ahead |
| verifier | Keeps the guesses the model agrees with |
| stream | Stop rules, the reply streamed back, crit serve and crit-eval |
Six slots every engine fills: model, placement, projections, attention, kv_cache and sampler. The others only when it needs them: a model without experts leaves the four expert slots open, and an engine without drafts leaves drafter and verifier open.
An engine fills the slots
An engine names one plugin for each slot it uses, in its configuration.json. Here is an example: crit Core with crit's library plugins in its slots and a model plugin of its own, for a dense model that drafts nothing.
- loadcrit Core
- placementdevice-placement
- requestcrit Core
- schedulecrit Core
- graphsrecorded-selection
- modelmy-model
- embeddingsmemory-table-rows
- normstorch-norms
- projectionsquantized-projections
- attentionsoftmax-attention
- kv_cacheattention-state
- recurrenceslot
- routerslot
- expertsslot
- expert_sourceslot
- expert_residencyslot
- constraincrit Core
- samplertoken-sampling
- drafterslot
- verifierslot
- streamcrit Core
{
"plugins": {"use": {
"placement": "device-placement",
"graphs": "recorded-selection",
"model": "my-model",
"embeddings": "memory-table-rows",
"norms": "torch-norms",
"projections": "quantized-projections",
"attention": "softmax-attention",
"kv_cache": "attention-state",
"sampler": "token-sampling"
}}
}Where plugins come from
- crit's library.
crit plugins addcopies a plugin into the engine as it is, like attention-state. - Adapted. The engine's copy of a library plugin with its own GPU code behind it, and the library plugin's tests still passing.
- Its own. Written by the engine, as every model plugin is, like
my-modelabove.
How Core checks an engine
- Each plugin alone, against a reference:
crit plugins test. - The plugins together, when the engine loads: every required slot filled, one plugin per slot, and what each plugin needs given by another plugin, the checkpoint or the GPU.
- The whole engine: one short request must reach every plugin, and
crit run ... ablatemeasures what each plugin adds.
crit-hub