crit-hub

request

A request comes in through crit serve or crit-eval and becomes a session of its own.

crit Core

What happens here

  1. The chat template turns the messages into the prompt's text, and the tokenizer turns that into tokens.
  2. Admission checks that the request fits: its context, its output limit and the memory for its keys and values.
  3. Each request gets its own session with its sampling settings and stop tokens.