crit-hub

Chunked prefill with a latency target

Execution A long prompt is read in chunks sized to keep each step under a latency target, through a cursor that moves on only when the model accepts a chunk.

In an engine

crit Core does this for every engine: it schedules each step's prefill chunk and batch tokens.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • Reading a long prompt in one go holds up every other request's next token until it is done.
  • A chunk can fail or be cancelled halfway, and a cursor that already moved on would skip those tokens or read them twice.
  • Later chunks cost more per token, since each token attends to everything before it, so no single chunk size suits the whole prompt.

How it works

  1. A cursor keeps a committed offset and at most one pending chunk. The pending chunk is handed out, and the committed offset stays where it was.
  2. The chunk is the largest whose estimated step time fits the target. The estimate adds a cost per call, a cost per token, attention over the earlier tokens and the decode rows of other requests sharing the step.
  3. The chunk runs as one or more physical calls to the model, each no larger than a cap of its own, set apart from the chunk size. Every call reuses one scratch buffer (working memory), sized for the call that needs the most and allocated once.
  4. When the model accepts the chunk, the committed offset moves to its end. If the step is cancelled or a call fails, the engine drops what the chunk wrote and the cursor rolls back, so the same tokens run again.

Pays off when

  • Long prompts arrive while other requests are decoding, and their time between tokens matters.
  • Long contexts with full attention, where later chunks cost much more per token than early ones.
  • A second model, such as a small draft model, needs smaller physical calls than the main one.

Doesn't pay off when

  • Every chunk and every call reads all the weights again, so chunks that are too small waste time.
  • Nothing else is waiting on the GPU, so a latency target only adds calls.
  • The estimate needs costs measured on the real engine; with bad ones, chunks miss the target or come out needlessly small.

What must stay true

  • The committed offset moves only when the model accepts the chunk, never when the chunk is handed out.
  • A rollback resets the cursor, and the engine also undoes the keys, values and other state the chunk wrote.
  • Physical calls run in order and cover exactly the chunk's tokens; a partly run chunk is never published.
  • Scratch is sized for every actual call shape and reused only after the previous call has finished.