Row compaction, metadata patches and retirement
Execution One step serves several requests: only ready rows join, results return by request identity, metadata is patched instead of rebuilt, and memory is freed only once the GPU is done.
In an engine
crit Core does this for every engine, in its concurrent step scheduler.
The problem
- Requests that share a step are at different points. Some still have GPU work running, and each brings its own number of candidate tokens, the new tokens the step proposes for it.
- A step's results sit in buffers the model can overwrite when it commits the step, and the per-request metadata the kernels read would otherwise be rebuilt every step.
- A finished request's memory may still be read by GPU work in flight, yet waiting for the whole batch would hold every early finisher back.
How it works
- Compaction takes the rows whose GPU work is done, in order, up to a row limit. Each compact row keeps its request id and generation (a number bumped when an id or slot is reused), so its result can only land on that request.
- Before the commit, which may overwrite the model's borrowed buffer, each row's candidate tokens are copied out of it and their bytes are charged to the memory budget.
- The commit runs once, and the group is marked used before it starts. Only when it returns successfully are the rows published to their requests.
- Per-request metadata lives in stable slots. The device copy gets a patch of only the rows that changed, and applies it only on the snapshot the patch was made from.
- A finished request leaves its slot, but a barrier holds its memory until every piece of GPU work tied to it reports that it has physically finished.
Pays off when
- Many requests share each step, and their GPU work finishes at different times.
- Speculative decoding, where each request accepts a different number of tokens per round.
- The metadata table is large and a step changes only a few of its rows.
Doesn't pay off when
- The engine runs one request at a time, or its own staging already does the same hand-off.
- Every step pays for scanning the rows and copying the tokens, which shows when steps are very short.
- Small groups leave the GPU underused, and waiting for fuller ones adds latency.
What must stay true
- A compact row keeps its request id and generation in every input, output and state update, so a reused slot can never make an old result current.
- Tokens are copied and charged before anything can overwrite the borrowed buffer, and nothing is published until the commit succeeds.
- Memory is freed when the GPU work that reads it has physically finished, never after a timeout.
- A patch applies only on its exact base snapshot, and older snapshots keep their bytes until their readers finish.
crit-hub