Glossary

The vocabulary you meet when running models locally, each linked to where it shows up in the data.

KV cache: Stored attention keys and values for tokens already processed, so they aren't recomputed each step.
GGUF: The single-file model format used by llama.cpp, carrying weights plus all metadata needed to run them.
Quantization: Storing weights at reduced precision to shrink a model, trading some quality for memory.
Sliding-window attention: Layers that attend only to a fixed recent window rather than the whole context.
MLA (multi-head latent attention): An attention variant that caches a compressed latent instead of full keys and values.
GQA (grouped-query attention): Multiple query heads share a smaller number of key/value heads, shrinking the cache.
Mixture of experts (MoE): Only a few of many feed-forward experts run per token, but all must be held in memory.
Active parameters: The subset of an MoE model's parameters involved in producing any single token.
Memory bandwidth: How fast a processor can read from memory — the main determinant of token generation speed.
Prefill (prompt processing): Processing the input prompt before generation starts. Compute-bound, unlike generation.
KV cache quantization: Storing the attention cache at reduced precision, roughly halving its size for little quality cost.
CPU offload: Keeping some layers in system RAM when a model does not fit entirely in VRAM.
Importance matrix (imatrix): Calibration data guiding which weights to preserve at higher precision during quantization.
Roofline model: Predicting performance from whichever resource saturates first — here, memory bandwidth.
Context window: The maximum number of tokens a model can attend to at once, including both your input and its output.
Tokens per second: How fast a model generates. Bound by memory bandwidth, not by arithmetic throughput.
Flash attention: An attention implementation that avoids materializing the full attention matrix, saving memory and time.
mmproj (multimodal projector): The separate file that lets a vision-language model actually see images.
Tensor parallelism: Splitting each layer across several GPUs so they compute together, rather than taking turns.
Sharding: Splitting one large model file into numbered parts. You need all of them.
Unified memory: One memory pool shared by CPU and GPU, as on Apple Silicon and some AMD systems.