crit-hub

memory-table-rows

Embedding table in memory

The embedding table is one tensor in memory, indexed by a step's token ids.

The animation needs JavaScript; How it works below says the same in words.

The problem

  • The first thing a model does with a token is look up its row in the embedding table, one row per vocabulary entry, often hundreds of thousands of them.
  • Some tables are larger than memory and are read from disk instead, and those must give exactly the rows the plain table gives.

How it works

  1. As the engine loads, the whole table is loaded into memory.
  2. A step asks for the rows of its token ids, and each id gets its row, in the order asked, repeated ids included.
  3. Its device form stores the table as one tensor on the device and indexes it with the step's ids.

Configuration

Its options, set per engine in configuration.json. Each is checked against its type and range when the engine loads.

OptionDefaultValuesWhat it sets
hostfalsetrue, falsedevice form: store() keeps the table in registered host memory, gathered over the link (lookup-table-rows' HostRows), not on the device
configuration.json
{"plugins": {
  "use":     {"embeddings": "memory-table-rows"},
  "options": {"memory-table-rows": {"host": false}}
}}

Pays off when

  • The table fits in memory beside the rest of the model, as most token embedding tables do.
  • A table read from disk is checked: lookup-table-rows is compared with it row by row.

Doesn't pay off when

  • The table is larger than the memory it would take, such as a large n-gram table, while each token reads only a few rows of it. lookup-table-rows keeps it on disk.

What must stay true

  • Every id gets exactly its own row, bit for bit.
  • Rows come back in the order they were asked for.