Batched serving
Execution Requests share each step: all their prompt and decode rows go through one matrix multiply per weight, and each request's output stays exactly what it would be alone.
In an engine
crit Core does this for every engine: its concurrent step scheduler puts many requests' rows in one step.
The problem
- Decoding one request at a time reads every weight of the model for a single token, so the GPU mostly waits on memory.
- Requests arrive at different times, and some are still reading their prompt while others decode, so their work never lines up by itself.
- If a request's numbers depend on which requests share its step, the same prompt gives different answers under load.
How it works
- Each step takes one row per decoding request (its last token) and a chunk of rows from each prompt still being read, decode rows first.
- Every projection is one matrix multiply (a GEMM) over all the step's rows, so the step reads each weight once, however many requests it holds.
- Attention sends decode rows and prompt rows to kernels of their own. Only rows that end a prompt or a decode go through the head to pick a token.
- Norms and activations read only their own row, and each weight's sums are split and added in an order fixed by its shape and the GPU, never by the step's row count. So a request's tokens are the same, bit for bit, alone or batched.
Pays off when
- Many requests are in flight at once, so each read of the weights serves many rows.
- Decode-heavy traffic, where a single request leaves the GPU waiting on memory.
- Answers must not change under load, for tests, evaluation or reproducible output.
Doesn't pay off when
- The engine serves one request at a time, so there is no one to share a step with.
- Steps are already so full that compute or attention, not reading the weights, sets the pace, so more requests make every step longer.
- Keeping output independent of the batch rules out some faster kernels, such as choosing how to split a GEMM by the step's row count.
What must stay true
- A row's arithmetic never depends on the other rows of its step or on where its prompt was cut into chunks.
- A row's key and value are written only to the cache page its step names, and before the attention that reads them.
- Admission reserves every page a request will need, prompt and output, so an admitted request can always finish.
- Pages a finished request gives back are reused only after the work that read them has settled.
crit-hub