Why long context eats your VRAM, and when it doesn't

The formula, and the term everyone gets wrong

Size your cache from num_key_value_heads, never num_attention_heads. Remember that llama.cpp allocates the whole thing at startup for the full -c you asked for. And before you cut your context, try -ctk q8_0 -ctv q8_0, which costs 8.5 bits per element instead of 16. For a conventional transformer:

bytes = 2 × n_layers × num_key_value_heads × head_dim × context_length × bytes_per_element

The 2 is K and V. Llama-3.1-8B has 32 layers, num_key_value_heads: 8, and no head_dim key at all, so you derive it as 4096/32 = 128. At 32,768 tokens in f16 that is exactly 4.00 GiB, on top of the weights, and it does not shrink when you switch to a Q4 quant of the model. Quantizing weights does nothing to the cache.

Linearity is the whole problem, and llama.cpp compounds it by allocating up front — the cache is sized at cparams.n_ctx_seq when the context is created, not grown as you fill it. -c 131072 costs 16 GiB on that model the moment the process starts, even if every prompt you send is 900 tokens.

Grouped-query attention is where the arithmetic usually goes wrong. Several query heads share one KV head, and only the KV heads are stored. Qwen3-8B declares num_attention_heads: 32 and num_key_value_heads: 8, and its GGUF carries qwen3.attention.head_count_kv = 8. Use the 32 and you are four times too high.

head_dim deserves the same suspicion. Deriving it as hidden_size / num_attention_heads works for Llama-3.1-8B and fails for Qwen3.6-27B, where hidden_size is 5120 and num_attention_heads is 24 — that division gives 213.33, while the config states head_dim: 256 outright. When the key is present, read it.

That formula is right for Llama, Mistral, and the Qwen3 dense models. It is wrong for three families that are now everywhere.

Sliding window: Gemma-3

Gemma-3-27B declares 62 layers, sliding_window: 1024, and sliding_window_pattern: 6. llama.cpp reads that period and calls set_swa_pattern(6), which marks a layer as windowed when il % 6 < 5. Full attention therefore lands on layers 5, 11, 17 … 59 — ten of them, leaving 52 windowed.

The windowed layers are not sized at exactly 1024. llama-kv-cache-iswa.cpp computes their capacity as:

uint32_t size_swa = GGML_PAD(std::min(size_base, hparams.n_swa*(unified ? n_seq_max : 1) + n_ubatch), 256);

With one sequence and the default 512-token ubatch that is 1024 + 512 = 1536 cells. With 16 KV heads at head_dim 128, each layer costs 8,192 bytes per cell, so at 32K:

• 10 full layers × 32,768 × 8,192 = 2.500 GiB
• 52 windowed layers × 1,536 × 8,192 = 0.609 GiB
• Total 3.109 GiB, against 15.50 GiB from the flat formula. Just under 5x.

--swa-full throws all of it away. The flag sets size_swa = size_base, giving all 62 layers a full 32K allocation and landing you on exactly the 15.50 GiB number. It exists for correctness when context is reused across slots in ways the windowed cache cannot reconstruct. If you pass it because a forum post said it fixed something, you are paying 5x for it.

MLA: DeepSeek-V3

Read DeepSeek-V3's config.json naively and it looks catastrophic: 61 layers, num_key_value_heads: 128, K width 192 (qk_nope_head_dim 128 + qk_rope_head_dim 64), V width 128. Stored per-head, 32K of context is 152.5 GiB.

Multi-head latent attention does not store it per-head. It caches one compressed latent of width kv_lora_rank (512) plus the shared RoPE part (64) — 576 total — and reconstructs the heads on the fly. llama.cpp's converter collapses this to MQA in one line, self.hparams["num_key_value_heads"] = 1, under the comment "deepseek2 using MLA converts into MQA (ie: GQA with 1 group)". The cache allocator then settles the V question:

const bool has_v = !is_mla;

No V tensor is created at all. At 32K in f16 the entire cache is 61 × 32,768 × 576 × 2 = 2.145 GiB. A 671B model has a smaller KV cache than Llama-3.1-8B, which is why DeepSeek-class models are viable on CPU-plus-RAM rigs where context, not weights, would otherwise be the wall.

The 152.5 GiB is not hypothetical, though, and this is the practical part. Conversions made before MLA support landed wrote the decompressed geometry into the GGUF, and those files are still on HuggingFace — a BF16 DeepSeek-V3-0324 conversion I read this session still carries head_count_kv = 128, key_length = 192, value_length = 128. A current conversion of V3.1 carries head_count_kv = 1, key_length = 576, value_length = 512. Same architecture, 71x apart in cache size. If a deepseek2 model reports n_head_kv = 128 in llama.cpp's load output, re-download it.

Hybrid linear attention: Qwen3.5 / Qwen3.6

Qwen3.6-27B has 64 layers and a layer_types array: 16 entries say full_attention, 48 say linear_attention. Only the 16 keep a growing KV cache. With 4 KV heads at head_dim 256, at 128K in f16:

16 × 131,072 × 2 × 4 × 256 × 2 = 8.00 GiB, against 32.00 GiB if you assume all 64 layers cache. Exactly 4x, because the ratio is just 64/16.

The linear layers keep a fixed-size recurrent state instead. The converter maps linear_key_head_dim to ssm.state_size (128) and linear_value_head_dim × linear_num_value_heads to ssm.inner_size (6144); llama.cpp sizes the state as the product, 786,432 f32 elements, so 3.00 MiB per layer and 144 MiB across all 48. The convolution state adds (4-1) × (6144 + 2×16×128) = 30,720 f32 per layer, another 5.6 MiB total. Both are per sequence, and both are the same size at 4K context as at the model's full 262K. Past a certain length the growing part of your memory is only a quarter of the model's depth.

The lever you control: cache dtype

-ctk and -ctv set the cache element type independently. The two worth using are q8_0 and q4_0, and their real cost is not 8 and 4 bits. Each block is 32 elements with an f16 scale: q8_0 is 2 + 32 = 34 bytes per 32 values (8.5 bits), q4_0 is 2 + 16 = 18 bytes (4.5 bits). Qwen3-8B, 36 layers, 8 KV heads, head_dim 128:

llama-server -m qwen3-8b-q4_k_m.gguf -c 131072 -ctk q8_0 -ctv q8_0

Two constraints, both enforced in source. Quantizing V requires flash attention — llama-context.cpp throws quantized V cache was requested, but this requires Flash Attention. Since flash_attn_type now defaults to AUTO this usually just works, but if you disabled FA, -ctv will refuse. And the per-layer n_embd_head_k / n_embd_head_v must be divisible by the 32-element block size, which is fine for the 128 and 256 you will normally meet.

On quality we have no measurement of our own. q8_0 is widely reported as near-lossless and is the setting we would reach for first; q4_0 on K is where people start reporting degradation on long-context recall, and the usual advice is to quantize V to q4_0 and leave K at q8_0. Treat both as the community's experience, not as something we ran perplexity sweeps to establish.

Computing your own, and checking it

Open config.json, read num_hidden_layers, num_key_value_heads, and head_dim, then check three keys before you multiply: sliding_window with a pattern, kv_lora_rank, and layer_types. Any of the three means you count only the layers that actually cache, and the flat formula becomes an upper bound rather than an answer.

Then stop trusting arithmetic, including ours. llama.cpp prints n_head_kv, n_embd_k_gqa, and n_embd_v_gqa per layer at load, and on cache creation prints the allocation with its cell count and the K and V sizes broken out separately — two lines for windowed models, one labelled non-SWA and one SWA, with the cell counts that produced the split. Start the server once at your target -c and read them. That is the number your card actually has to hold.