crit-hub

expert-tensors

Expert weights in memory

Expert weights are held in memory as the checkpoint gives them. The plain source every other expert source is compared with.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A mixture-of-experts layer holds many experts, each a gate/up matrix and a down matrix, and the experts plugin needs an expert's matrices when a token is routed to it.
  • Where those matrices come from (memory, a file on disk) is a slot of its own, so the experts plugin computes the same whichever source an engine uses.

How it works

  1. As the engine loads, each expert's gate/up and down matrices are added to the source, layer by layer.
  2. It keeps them as they were added: no file, no encoding, no eviction.
  3. When the experts plugin asks for an expert's matrices, it gets those same matrices back.
  4. It provides experts:source, the fact an experts plugin needs before it can run.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
store_path""strthe file of the bundle store (built once, reopened while its layout holds); empty: anonymous host memory
csf_scales""strscale components of the experts plugin's slot layout the store keeps CSF-compressed, with the exception words each bundle holds for each ("gate_up_scale:256,down_scale:128"; ModelOpt's encoding, about half the bytes of E4M3 scales): put() encodes them from the slot-layout bundle or takes the checkpoint's (compressed=), the device expert cache rebuilds the slot layout as it fills slots (no CPU tier over such a store); empty: the slot layout as it is
configuration.json
{"plugins": {
  "use":     {"expert_source": "expert-tensors"},
  "options": {"expert-tensors": {"store_path": "", "csf_scales": ""}}
}}

Pays off when

  • Every expert fits in memory, so nothing has to be read from a file.
  • Another expert source or an experts plugin is tested: they are compared on the weights it holds.

Doesn't pay off when

  • The experts are larger than memory. ssd-expert-materialization keeps them in a file and reads each one exactly when it is needed.

What must stay true

  • An expert's matrices come back exactly as they were added, bit for bit.
  • The source holds weights and computes nothing: what a layer does with them is the experts plugin's.