crit-hub

schedule

Core's scheduler puts the rows of many requests into one step, so the GPU reads the weights once for all of them.

crit Core

What happens here

  1. Each step mixes prompt chunks of new requests with one decode row for each running request.
  2. The model plugin says how large a prompt chunk and a step may be; Core picks which rows go in.
  3. Requests join and leave between steps, so none waits for another to finish.

Learn more

How batching works, step by step.