expert-tensors
Expert weights in memory
Expert weights are held in memory as the checkpoint gives them. The plain source every other expert source is compared with.
The problem
- A mixture-of-experts layer holds many experts, each a gate/up matrix and a down matrix, and the experts plugin needs an expert's matrices when a token is routed to it.
- Where those matrices come from (memory, a file on disk) is a slot of its own, so the experts plugin computes the same whichever source an engine uses.
How it works
- As the engine loads, each expert's gate/up and down matrices are added to the source, layer by layer.
- It keeps them as they were added: no file, no encoding, no eviction.
- When the experts plugin asks for an expert's matrices, it gets those same matrices back.
- It provides
experts:source, the fact an experts plugin needs before it can run.
Configuration
Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.
| Option | Default | Values | What it sets |
|---|---|---|---|
store_path | "" | str | the file of the bundle store (built once, reopened while its layout holds); empty: anonymous host memory |
csf_scales | "" | str | scale components of the experts plugin's slot layout the store keeps CSF-compressed, with the exception words each bundle holds for each ("gate_up_scale:256,down_scale:128"; ModelOpt's encoding, about half the bytes of E4M3 scales): put() encodes them from the slot-layout bundle or takes the checkpoint's (compressed=), the device expert cache rebuilds the slot layout as it fills slots (no CPU tier over such a store); empty: the slot layout as it is |
configuration.json
{"plugins": {
"use": {"expert_source": "expert-tensors"},
"options": {"expert-tensors": {"store_path": "", "csf_scales": ""}}
}}Pays off when
- Every expert fits in memory, so nothing has to be read from a file.
- Another expert source or an experts plugin is tested: they are compared on the weights it holds.
Doesn't pay off when
- The experts are larger than memory.
ssd-expert-materializationkeeps them in a file and reads each one exactly when it is needed.
What must stay true
- An expert's matrices come back exactly as they were added, bit for bit.
- The source holds weights and computes nothing: what a layer does with them is the experts plugin's.
crit-hub