Chunked prefill with a latency target
Execution A long prompt is read in chunks sized to keep each step under a latency target, through a cursor that moves on only when the model accepts a chunk.
In an engine
crit Core does this for every engine: it schedules each step's prefill chunk and batch tokens.
The problem
- Reading a long prompt in one go holds up every other request's next token until it is done.
- A chunk can fail or be cancelled halfway, and a cursor that already moved on would skip those tokens or read them twice.
- Later chunks cost more per token, since each token attends to everything before it, so no single chunk size suits the whole prompt.
How it works
- A cursor keeps a committed offset and at most one pending chunk. The pending chunk is handed out, and the committed offset stays where it was.
- The chunk is the largest whose estimated step time fits the target. The estimate adds a cost per call, a cost per token, attention over the earlier tokens and the decode rows of other requests sharing the step.
- The chunk runs as one or more physical calls to the model, each no larger than a cap of its own, set apart from the chunk size. Every call reuses one scratch buffer (working memory), sized for the call that needs the most and allocated once.
- When the model accepts the chunk, the committed offset moves to its end. If the step is cancelled or a call fails, the engine drops what the chunk wrote and the cursor rolls back, so the same tokens run again.
Pays off when
- Long prompts arrive while other requests are decoding, and their time between tokens matters.
- Long contexts with full attention, where later chunks cost much more per token than early ones.
- A second model, such as a small draft model, needs smaller physical calls than the main one.
Doesn't pay off when
- Every chunk and every call reads all the weights again, so chunks that are too small waste time.
- Nothing else is waiting on the GPU, so a latency target only adds calls.
- The estimate needs costs measured on the real engine; with bad ones, chunks miss the target or come out needlessly small.
What must stay true
- The committed offset moves only when the model accepts the chunk, never when the chunk is handed out.
- A rollback resets the cursor, and the engine also undoes the keys, values and other state the chunk wrote.
- Physical calls run in order and cover exactly the chunk's tokens; a partly run chunk is never published.
- Scratch is sized for every actual call shape and reused only after the previous call has finished.
crit-hub