Memory bandwidth
How fast a processor can read from memory — the main determinant of token generation speed.
Generating one token requires reading the participating weights from memory. Generation speed is therefore roughly bytes-read-per-token divided by achievable bandwidth, which is why bandwidth predicts local inference speed far better than teraflops.
Real accelerators achieve a high but not complete fraction of their theoretical figure — around 93% on Apple's Metal backend and around 79% on consumer CUDA in fitted benchmarks — plus a fixed per-token overhead that becomes dominant on very fast memory.