dense-layer-offload
Layer offload for dense models
A dense model too big for the GPU keeps some layers there and, for the rest, runs them on the CPU or streams their weights in, whichever plan measures faster.
The problem
- A dense model reads every layer's weights for every token, unlike a mixture of experts. When those weights are larger than the GPU memory left for them, some layers have to stay in host memory.
- Those layers can run on the CPU or be copied to the GPU every step. Which is faster depends on this machine's GPU, CPU and the link between them.
How it works
- The engine measures this machine's rates. These are how fast the GPU reads its memory, how fast weights upload over PCIe, and how fast the CPU runs a matrix-vector product.
- The planner fills the GPU's budget with the layers that save the most time per byte. For the rest it predicts a step under two plans, running them on the CPU or streaming them through two GPU staging slots.
- A CPU layer runs from host memory, with its KV cache in host memory too. The activations go down to the host once before a run of CPU layers and back up after it.
- A streamed layer uploads into one of the two slots while earlier layers compute. Each upload waits only for the layer two back to free its slot, so copying and computing overlap (double buffering).
- Predictions only rank the plans. The engine times the leading ones for a few real steps and keeps the one that measured fastest.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
device_gbs | 0 | float, at least 0 | the device's measured bandwidth (crit eval probe), 0 when unmeasured |
upload_gbs | 0 | float, at least 0 | the measured pinned host-to-device bandwidth, 0 when unmeasured |
host_gbs | 0 | float, at least 0 | the measured host GEMV rate in bytes per ns, 0 when unmeasured |
host_layer_ns | 0 | float, at least 0 | the measured activation round trip of a run of host layers |
configuration.json
{"plugins": {
"use": {"placement": "dense-layer-offload"},
"options": {"dense-layer-offload": {"device_gbs": 0, "upload_gbs": 0, "host_gbs": 0}}
}}Pays off when
- The weights do not fit the GPU but fit host memory, and decode runs one sequence or a few rows per step.
- The CPU has wide vector units (AVX2 or AVX-512) or the link to the GPU is fast, so one of the two plans runs well.
Doesn't pay off when
- The weights fit the GPU, or a quantized form of them fits, stays within the quality budget and measures faster.
- The model is a mixture of experts, whose misses come per expert rather than per layer and are handled by expert caching and dispatch.
What must stay true
- A streamed layer is read only after its upload has landed, and its slot is reused only after that layer has released it.
- CPU layers read the activations only after the copy down has finished, and the GPU reads them only after the copy back.
- A layer's KV cache lives where the layer runs, so moving a layer starts a new sequence or moves its cache along.
- Plans are ranked by prediction and chosen by measurement.
crit-hub