ssd-expert-materialization
File-backed experts with read-ahead
Expert weights stay in a file on disk instead of memory: each expert is read as an exact, checked byte range, and the ones a predictor expects are read ahead.
The problem
- A mixture-of-experts model can have more expert weights than host and GPU memory hold together, so some of them can only stay in a file on disk.
- Reading from disk is slow next to memory, and a layer cannot run until the bytes of its experts have arrived.
- A short read, or a file that changed on disk, would hand the model wrong weights without any error.
How it works
- Each expert is a region of a read-only weights file, described by the model's metadata: an offset, a length, an encoding and a checksum.
- Experts that were not read ahead are demand reads, and the layer waits for them. One gather reads them all: it hints their ranges to the operating system, then a few workers copy the bytes into staging memory, in the order asked.
- Every read is checked: its length and its checksum must match the region, or the read is rejected.
- The checked bytes go to the upload. Staging keeps them until the upload reports done, which is a separate event from the read finishing.
- Read-ahead gathers the experts a predictor expects before the layer asks, with one hint for neighbouring regions, and a repeated request gets the pending read back. A guess is only read; its upload waits until a layer asks for it.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
host_bytes | 1048576 | int, at least 0 | Bytes of the host tier in front of the file |
directory | "" | str | Folder of the expert file |
readers | 16 | int 1 to 256 | threads reading bundles from the store file |
csf_scales | "" | str | scale components of the experts plugin's slot layout the store keeps CSF-compressed, with the exception words each bundle holds for each ("gate_up_scale:256,down_scale:128"; ModelOpt's encoding, about half the bytes of E4M3 scales): put() encodes them from the slot-layout bundle or takes the checkpoint's (compressed=), the device expert cache rebuilds the slot layout as it fills slots (no CPU tier over such a store); empty: the slot layout as it is |
configuration.json
{"plugins": {
"use": {"expert_source": "ssd-expert-materialization"},
"options": {"ssd-expert-materialization": {"host_bytes": 1048576, "directory": "", "readers": 16}}
}}Pays off when
- The model is larger than memory, so experts are served from a file instead of being held in RAM.
- A predictor names the next experts early enough for their reads to finish first, without crowding the disk, staging memory or the upload.
Doesn't pay off when
- The weights already fit in memory, or reading them is not what the engine waits on.
- Guesses are often wrong, so read-ahead spends disk time and staging memory on bytes nobody uses.
- Checksums cost a pass over every byte read, so an engine that skips them must trust the file another way.
What must stay true
- An expert is read exactly and completely, from a region in trusted metadata, and the file does not change while it is read.
- Staging keeps an expert's bytes until every reader, including the upload, has finished with them.
- A finished read is not a finished upload. The layer reads an expert only after its upload reports done.
- Read-ahead is only a guess. It never changes which expert runs, and a hint never counts as a finished read.
crit-hub