crit-hub

ssd-expert-materialization

File-backed experts with read-ahead

Expert weights stay in a file on disk instead of memory: each expert is read as an exact, checked byte range, and the ones a predictor expects are read ahead.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A mixture-of-experts model can have more expert weights than host and GPU memory hold together, so some of them can only stay in a file on disk.
  • Reading from disk is slow next to memory, and a layer cannot run until the bytes of its experts have arrived.
  • A short read, or a file that changed on disk, would hand the model wrong weights without any error.

How it works

  1. Each expert is a region of a read-only weights file, described by the model's metadata: an offset, a length, an encoding and a checksum.
  2. Experts that were not read ahead are demand reads, and the layer waits for them. One gather reads them all: it hints their ranges to the operating system, then a few workers copy the bytes into staging memory, in the order asked.
  3. Every read is checked: its length and its checksum must match the region, or the read is rejected.
  4. The checked bytes go to the upload. Staging keeps them until the upload reports done, which is a separate event from the read finishing.
  5. Read-ahead gathers the experts a predictor expects before the layer asks, with one hint for neighbouring regions, and a repeated request gets the pending read back. A guess is only read; its upload waits until a layer asks for it.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
host_bytes1048576int, at least 0Bytes of the host tier in front of the file
directory""strFolder of the expert file
readers16int 1 to 256threads reading bundles from the store file
csf_scales""strscale components of the experts plugin's slot layout the store keeps CSF-compressed, with the exception words each bundle holds for each ("gate_up_scale:256,down_scale:128"; ModelOpt's encoding, about half the bytes of E4M3 scales): put() encodes them from the slot-layout bundle or takes the checkpoint's (compressed=), the device expert cache rebuilds the slot layout as it fills slots (no CPU tier over such a store); empty: the slot layout as it is
configuration.json
{"plugins": {
  "use":     {"expert_source": "ssd-expert-materialization"},
  "options": {"ssd-expert-materialization": {"host_bytes": 1048576, "directory": "", "readers": 16}}
}}

Pays off when

  • The model is larger than memory, so experts are served from a file instead of being held in RAM.
  • A predictor names the next experts early enough for their reads to finish first, without crowding the disk, staging memory or the upload.

Doesn't pay off when

  • The weights already fit in memory, or reading them is not what the engine waits on.
  • Guesses are often wrong, so read-ahead spends disk time and staging memory on bytes nobody uses.
  • Checksums cost a pass over every byte read, so an engine that skips them must trust the file another way.

What must stay true

  • An expert is read exactly and completely, from a region in trusted metadata, and the file does not change while it is read.
  • Staging keeps an expert's bytes until every reader, including the upload, has finished with them.
  • A finished read is not a finished upload. The layer reads an expert only after its upload reports done.
  • Read-ahead is only a guess. It never changes which expert runs, and a hint never counts as a finished read.