crit-hub

Host memory admission

Memory and loading Before a large host allocation, the engine checks what the operating system reports it can back (free memory, the commit limit, the container's cgroup), not only its own budget.

In an engine

No slot holds it: a model plugin calls it before a large host allocation.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • An engine's memory budget counts only its own reservations. It cannot see other processes, the container's limit or what the OS can actually back.
  • An allocation the budget allows can still make the host page to disk, or hit the commit limit, the total memory the OS will promise to all processes.
  • In a container it can also get the process throttled or killed by its cgroup, the kernel's memory limit for a group of processes.

How it works

  1. Before a large host allocation, the engine reads three observations from the OS. They are physical memory available, commit headroom and the cgroup's headroom.
  2. An observation the OS does not report or enforce stays unknown, with its reason, and never counts as zero. Commit headroom counts only where the OS enforces a commit limit.
  3. cgroup headroom is the limit minus usage plus clean file cache, which the kernel can drop at once. Pages not yet written to disk (dirty or under writeback) and shmem, memory that lives only in RAM, do not count. The tightest cgroup level wins.
  4. The request plus a reserve for the OS and other processes must fit the tightest known observation. A denial names that binding limit.
  5. On a denial the engine shrinks its plan, for example to smaller chunks or fewer resident items, or fails the request. When admitted, it allocates and then charges its budget.

Pays off when

  • The engine shares the host with other work, or runs in a container with a memory limit.
  • It makes large host allocations, such as pinned staging, resident weights or a CPU KV cache.

Doesn't pay off when

  • The engine owns a dedicated machine with measured headroom, or its allocations are small next to free memory.
  • An observation is only a snapshot. Another process can take the memory before the allocation, so an admitted request can still fail.

What must stay true

  • An unknown observation stays unknown with a reason, never zero, and with nothing known, admission denies by default.
  • Only clean file cache counts as room; dirty, writeback and shmem (tmpfs) pages never do.
  • The request plus an explicit reserve is compared with the tightest known observation.
  • The allocation's own result stays the authority. Admission is evidence, not a reservation.