placement
Where each tensor lives at the context the engine serves (the device, host memory or files), decided before anything loads. The plan is the placement, not advice: on CUDA Core's loader refuses an engine that holds more device memory than its plan put there.
requiredhost policy
In crit's library
Every tensor on the deviceEvery tensor goes on the device, with room kept free beside the plan; a model that does not fit is refused before anything loads.placementLayer offload for dense modelsA dense model too big for the GPU keeps some layers there and, for the rest, runs them on the CPU or streams their weights in, whichever plan measures faster.placement
Interface
The methods the engine calls on the slot's plugin, in their host form: token ids, rows of numbers and dicts. Every plugin of the slot has them, which is what lets an ablation switch one for another.
| Method | What it does |
|---|---|
plan(tensors, budget) | Where each tensor lives: device, host or files, within the bytes free at the served context |
crit-hub