Host work mailbox
Execution From inside a captured CUDA graph, the GPU posts work for the host into a shared mailbox, and the stream waits for the answer only where it reads it.
In an engine
No slot holds it: a model plugin calls it from inside its captured graphs.
The problem
- A CUDA graph records a step's GPU work once and replays it every step without the CPU launching anything. Some work only the host can do, such as computing an expert the GPU does not hold.
- Which host work a step needs is only known on the GPU, in the middle of the step. Splitting the graph there and synchronizing with the host costs a round trip at every such point.
How it works
- Before capture, the engine times two ways for the stream to wait for the host, a wait held by the GPU's front end and a one-thread kernel that polls. The cheaper one goes into the graph.
- Inside the graph, a kernel decides that a slot needs the host. It writes the job into the slot, in pinned host memory the GPU can map, sets done to 0 and then rings the doorbell by setting ready to 1.
- A slot with no host work this time is answered by the GPU itself, as skipped with done set to 1, so every wait in the graph still completes.
- A host service thread sees the doorbell, runs the job, writes the result and the status, and only then sets done to 1. The stream keeps working meanwhile and waits for done only at the join, right before it reads the result.
- The next replay runs the same graph, whose join waits for the same value fixed at capture. That is safe because every post resets done before it rings.
Pays off when
- A graph-captured step needs host work that the GPU decides on, such as experts computed on the CPU, host table lookups or copies the host issues.
- The GPU has other work to do between the post and the join, so the host's time hides behind it. Where the driver makes the front-end wait cheap, it also leaves every compute unit free for other kernels.
Doesn't pay off when
- The host work is known before launch, so it can simply be queued ahead, or it fits on the GPU.
- The join comes right after the post with nothing to overlap, and the measured wait costs more than the work it moves.
- No CPU core can be spared for the service thread, which spins while jobs arrive.
What must stay true
- The GPU fences the job, resets done and only then rings ready. Only the GPU runs on every replay, so the reset is never left to the host.
- The host writes the status before done, and done reaching 1 is the only sign that a job is finished.
- A slot holds one job at a time until the stream has joined it, and a slot without work is answered by the GPU, so every captured join completes.
- A released or failed job is still answered, so a stuck step always drains, and its outputs are thrown away. After a handler fails, the mailbox answers every later job as failed.
crit-hub