crit-hub

eager-graphs

Every step eager

Every step runs as eager kernel launches: no capture, nothing to choose. The plain choice the graphs slot falls back to.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A step can run eagerly, with the host launching each kernel, or replay a captured CUDA graph of it. Replaying saves the launches, but a captured graph is tied to its shapes and buffers.
  • The graphs slot picks the form each step runs in. One choice has to be the plain one that captures nothing, to bring a model up on and to compare captures with.

How it works

  1. For each step, the model plugin offers the forms it can run: eager, and graphs captured for that phase and shape.
  2. This plugin takes the first eager offer whose shape and identity match the step exactly.
  3. It never takes a captured graph, so the host launches every kernel of every step.
  4. It measures nothing and records nothing, so it picks the same on every machine.

Configuration

No options: an engine selects it, and that is the whole setting.

configuration.json
{"plugins": {
  "use": {"graphs": "eager-graphs"}
}}

Pays off when

  • A new model plugin is being brought up, and every step should first run the plain way.
  • Steps are large, as in long prompts, so launches are a small part of each step's time.

Doesn't pay off when

  • Decode runs small batches, where launches take a large share of a step and a replayed graph is faster. recorded-selection picks a graph where one measures faster.

What must stay true

  • A step runs only an offer that matches its exact shape and identity.
  • Nothing is timed, so every process of the engine runs every step the same way.