Prefill (prompt processing)
Processing the input prompt before generation starts. Compute-bound, unlike generation.
Prefill runs the whole prompt through the model in parallel, so it is limited by arithmetic throughput rather than memory bandwidth. This produces a counterintuitive result: quantizing a model does not speed up prompt processing, and often slightly slows it, because weights must be dequantized. Generation, being bandwidth-bound, does get substantially faster.
It is why a machine can feel fast at generating and slow to start on a long document.