eager-graphs
Every step eager
Every step runs as eager kernel launches: no capture, nothing to choose. The plain choice the graphs slot falls back to.
The problem
- A step can run eagerly, with the host launching each kernel, or replay a captured CUDA graph of it. Replaying saves the launches, but a captured graph is tied to its shapes and buffers.
- The graphs slot picks the form each step runs in. One choice has to be the plain one that captures nothing, to bring a model up on and to compare captures with.
How it works
- For each step, the model plugin offers the forms it can run: eager, and graphs captured for that phase and shape.
- This plugin takes the first eager offer whose shape and identity match the step exactly.
- It never takes a captured graph, so the host launches every kernel of every step.
- It measures nothing and records nothing, so it picks the same on every machine.
Configuration
No options: an engine selects it, and that is the whole setting.
configuration.json
{"plugins": {
"use": {"graphs": "eager-graphs"}
}}Pays off when
- A new model plugin is being brought up, and every step should first run the plain way.
- Steps are large, as in long prompts, so launches are a small part of each step's time.
Doesn't pay off when
- Decode runs small batches, where launches take a large share of a step and a replayed graph is faster.
recorded-selectionpicks a graph where one measures faster.
What must stay true
- A step runs only an offer that matches its exact shape and identity.
- Nothing is timed, so every process of the engine runs every step the same way.
crit-hub