crit-hub

Checked tensor loading

Memory and loading Each tensor is read from the checkpoint file at its exact byte range, checked against its digest, and copied with both ends kept alive until the copy is done.

In an engine

No slot holds it: a model plugin calls it to load the checkpoint's tensors.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • A checkpoint holds many tensors at fixed byte positions, and an engine often loads only some of them, one at a time or in its own order.
  • A size computed from shape and element type is wrong for packed formats with separate scales, and a wrong size silently reads the next tensor's bytes.
  • Copies to the GPU run in the background. Freeing the source too early, or reading a file that changed on disk, gives wrong weights without any error.

How it works

  1. The engine's parser gives a manifest. For each tensor it lists an offset and a byte count, a shape and an encoding, and a SHA-256 digest of its bytes.
  2. Opening the file checks every region. Each must lie inside the file and no two may overlap; sizes come from the file's header, never from shape times element size.
  3. A read takes exactly the region's bytes and hashes them. A short read or a different digest is refused.
  4. The copy to the GPU gets a fresh destination. The staging bytes, the destination and the copy's readiness stay together and alive until the copy is done, and only then is the tensor published.
  5. A large region is read in fixed chunks in file order. Each chunk is hashed and copied out while the next is read, and the tensor is published only when the digest over all chunks matches.

Pays off when

  • The checkpoint stores tensors that can be addressed one by one, and the engine loads only some of them, lazily or in its own placement.
  • Weights go to the GPU through host staging in the background, while the next region is already being read.

Doesn't pay off when

  • The engine's loader already checks exact extents, content and copy lifetimes, so a second read-and-hash pass only adds time.
  • Every byte read is also hashed, which can dominate many small reads.
  • Staging needs host memory on top of the destination, so it has to be budgeted and admitted too.

What must stay true

  • A region's offset and size describe where its bytes are; shape and encoding only say how to read them, and neither is computed from the other.
  • Both ends of a copy, the source bytes and the destination, stay alive until the copy has really finished, even when submitting it failed.
  • Only bytes whose digest matches are published, so a changed or truncated file is refused instead of loaded.