schedule
Core's scheduler puts the rows of many requests into one step, so the GPU reads the weights once for all of them.
crit Core
What happens here
- Each step mixes prompt chunks of new requests with one decode row for each running request.
- The model plugin says how large a prompt chunk and a step may be; Core picks which rows go in.
- Requests join and leave between steps, so none waits for another to finish.
Learn more
How batching works, step by step.
Batched serving
Requests share each step: all their prompt and decode rows go through one matrix multiply per weight, and each request's output stays exactly what it would be alone.
Row compaction, metadata patches and retirement
One step serves several requests: only ready rows join, results return by request identity, metadata is patched instead of rebuilt, and memory is freed only once the GPU is done.
Chunked prefill with a latency target
A long prompt is read in chunks sized to keep each step under a latency target, through a cursor that moves on only when the model accepts a chunk.
crit-hub