crit-hub

recorded-selection

Recorded execution choices

For each phase and shape, the engine picks eager launches, a full graph or a partial graph from measured costs, and records the pick so later processes repeat it.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A step can run eagerly, with the CPU launching each kernel, or replay a captured CUDA graph of all or part of it. Which is fastest depends on the phase, the shape, how often it repeats and the memory left.
  • Picking by timing at every load is slow and noisy. Interchangeable kernels round differently, so a near tie that flips makes two processes of the same engine write different text.

How it works

  1. For each phase and shape the engine has offers, such as running eagerly, replaying a full captured graph or replaying a partial one. Each comes with a measured run time, capture time and memory need.
  2. Offers that cannot run here are set aside with a reason, such as a capture made for other buffers or more memory than is free.
  3. The rest are ranked by total cost, the run time multiplied by the expected uses plus the capture time if the graph is not captured yet. Kernel variants of one operation are tuned the same way, and a tie keeps the lower index.
  4. The tuning run, done once in development, writes every choice into the engine's frozen configuration, with the library version and device it was tuned on.
  5. Every serving process replays the recorded choices without timing anything, and a repeatability check runs fresh processes and compares their complete outputs.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
modeseager,full,partialstrCaptures it may choose from: eager, full graphs, partial graphs
allow_paddingfalsetrue, falseLet a step run in a larger captured shape, padded
configuration.json
{"plugins": {
  "use":     {"graphs": "recorded-selection"},
  "options": {"recorded-selection": {"modes": "eager,full,partial", "allow_padding": false}}
}}

Pays off when

  • Several ways to run a step or an operation exist, and which is fastest changes with the phase and the shape.
  • Decode shapes repeat thousands of times, so a capture is paid once and saves launches on every step.
  • Greedy output must be the same in every process, for an exactness or repeatability check.

Doesn't pay off when

  • Only one way to run exists, or every candidate gives bit-identical results for the engine's inputs.
  • The offers lack trustworthy measured costs, or a complete record of the buffers and backend each was captured for.
  • The library or the device changes often. Each change needs a new tuning run, because an old table can point at a different candidate.

What must stay true

  • Nothing is timed while serving. A recorded choice replays exactly, and an index beyond the candidate list is an error.
  • The candidate list keeps the same order for the recorded library version and device.
  • An offer runs only if its phase, shape and identity (the buffers and backend it was captured for) match and its memory still fits. A recorded index proves neither.
  • A choice recorded for a range of widths is used at a new width only if the backend accepts it for that exact shape.