Mixture of experts (MoE)
Only a few of many feed-forward experts run per token, but all must be held in memory.
An MoE model routes each token to a small subset of its experts. A 120B model with 128 experts and top-4 routing reads only about 3% of its expert weights per token, but every expert must still be resident.
Resident size and per-token bytes are therefore different questions with very different answers, and collapsing them into one number — as a single "VRAM required" figure does — is misleading in both directions. It also makes MoE unusually well suited to CPU offload, since the bulk of the bytes are read sparsely.