crit-hub

device-placement

Every tensor on the device

Every tensor goes on the device, with room kept free beside the plan; a model that does not fit is refused before anything loads.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • An engine's tensors (its weights, its KV cache at the context it serves, any recurrent state) each live somewhere: on the device, in host memory or in files. Where has to be decided before anything loads.
  • A wrong plan shows up late, as a load that runs out of device memory halfway, or a long request that fails once its KV cache has grown.

How it works

  1. Before any weight is read, Core's loader asks the placement for its plan at the context length the engine serves.
  2. This plan puts every tensor on the device: the weights, the KV cache for the full served context and any recurrent state.
  3. The budget is the device's free memory less reserve_mib, room kept for the CUDA context, workspaces, graph pools or a desktop on the same GPU.
  4. A model whose tensors do not fit the budget is refused before anything loads.
  5. On CUDA the loader then checks that the engine holds no more device memory than the plan put there.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
reserve_mib0int 0 to 65536MiB of the device free memory kept out of the plan (CUDA context, workspaces, graph pools, a desktop on the same GPU)
off_device""strcomma-separated name prefixes of groups that live off the device, in order of preference for host memory ("experts.,tables.": routed experts' bundles, then lookup tables); empty: every group on the device
filestruetrue, falseoff-device groups that host memory cannot hold go to files (false: refused)
configuration.json
{"plugins": {
  "use":     {"placement": "device-placement"},
  "options": {"device-placement": {"reserve_mib": 0, "off_device": "", "files": true}}
}}

Pays off when

  • The weights, the KV cache at the served context and any state all fit in the device's memory.
  • A placement that keeps tensors in host memory or files is compared with the plain one, on a model that fits.

Doesn't pay off when

  • The model is larger than the device's memory. A placement that keeps part of it in host memory, such as dense-layer-offload, can serve it, more slowly.
  • Something else on the GPU changes how much memory it takes while the engine runs: the reserve is fixed when the engine loads.

What must stay true

  • The plan is the placement, not advice: the model plugin keeps on the device only what the plan puts there.
  • A model that does not fit is refused before anything loads, never half loaded.
  • The reserve stays outside the plan.