CUDA out of memory

Almost everyone sizing a model counts the weights and stops. Three other terms compete for the same memory: the KV cache, which grows linearly with context and frequently exceeds the weights past 32K tokens; the compute buffer, which scales with batch size and vocabulary; and roughly half a gigabyte of CUDA context that exists before any of your model loads. A file that is 2 GB smaller than your card still fails when the cache is 4 GB.

1. 1
   Shorten the context

   The cache is allocated up front for the full context you request, not grown as you fill it. Asking for 128K when your prompts are 4K costs you the full 128K allocation immediately.

2. 2
   Quantize the KV cache

   --cache-type-k q8_0 --cache-type-v q8_0 roughly halves the dominant term at long context for very little quality cost. It is the most useful setting most people never change.

3. 3
   Drop one quantization level

   Q5_K_M to Q4_K_M is a small quality step and a real memory saving. Going below Q3 is where quality falls off sharply for most models.

4. 4
   Offload experts, if it's a mixture-of-experts model

   --n-cpu-moe N moves expert tensors to system RAM. Experts are the bulk of the bytes but only a few are read per token, so this costs far less speed than offloading dense layers.

5. 5
   Offload layers

   -ngl N keeps only N layers on the GPU. Recent llama.cpp defaults to -ngl auto with --fit on and will size this for you.

If free VRAM looks sufficient and it still fails, check for another process holding memory, and on Windows account for the desktop compositor's reserve. Fragmentation can also block a large contiguous allocation when the total free figure looks fine.

Rather than guessing, pick your model and card and read the grid: every quantization at every context, with the memory each combination actually needs.