request
A request comes in through crit serve or crit-eval and becomes a session of its own.
crit Core
What happens here
- The chat template turns the messages into the prompt's text, and the tokenizer turns that into tokens.
- Admission checks that the request fits: its context, its output limit and the memory for its keys and values.
- Each request gets its own session with its sampling settings and stop tokens.
crit-hub