Choosing a GGUF quantization

Start here

Take the largest quant that still leaves room for your KV cache. Q4_K_M when memory is tight, Q5_K_M or Q6_K when it isn't, IQ3_M or IQ2_M when you are forcing a model into a card it doesn't fit. Below about 4 bits per weight, prefer the IQ variants over the Q*_K ones at equal file size; at 4 bits and above the two are a wash.

That advice is deliberately generic, because whether Q4_K_M is good enough for your model is a question nobody can currently answer with data. Here is what the labels mean and where the evidence stops.

A quant label names a mixture, not a precision

Q4_K_M does not mean "every weight is 4 bits." It names a recipe that assigns different types to different tensors. Read out of the header of bartowski/Qwen_Qwen3-8B-GGUF/Qwen_Qwen3-8B-Q4_K_M.gguf (399 tensors):

A quarter of the file is not 4-bit. The Q6_K tensors are output.weight plus attn_v and ffn_down on 18 of the 36 layers, and which 18 is not arbitrary: llama.cpp's use_more_bits selects i < n/8 || i >= 7n/8 || (i - n/8) % 3 == 2, which for 36 layers gives 0–3, every third layer from 6, and 31–35. Ends get more bits, the middle gets sampled. Every norm tensor stays F32 in every quant, including Q8_0.

So effective bits per weight always exceeds the label, by a model-dependent amount. A Q4_K block is 144 bytes per 256 weights, exactly 4.5 bpw. If every quantizable tensor in Qwen3-8B were Q4_K the tensor data would be 4,608,348,160 bytes; the real file is 5,027,784,224 — 9.0% more, an effective 4.911 bpw.

The gap closes at the top — Qwen3-8B's Q6_K is 254 Q6_K tensors and 145 F32 norms, nothing else — and blows out at the bottom, where Q2_K runs 22% over its name because the recipe pushes attn_v to Q4_K and ffn_down and attn_output to Q3_K. Two-bit versions of those tensors wreck the model. Vocabulary shifts it too: Qwen3-8B's 151,936-token vocab makes token_embd plus output 15.2% of the parameters, and output stays Q6_K. The same recipe on 32k-vocab Mistral-7B lands near 4.83 bpw.

Suffixes

_S and _M differ only in how generously the recipe promotes tensors. Q4_K_S promotes attn_v and ffn_down to Q5_K on layers 0–3 only; Q4_K_M promotes them to Q6_K on 18 layers. That is the whole difference: 4.690 vs 4.911 bpw.

Q3_K_L is a real upstream type. Q4_K_L, Q5_K_L, Q6_K_L, Q2_K_L, Q3_K_XL are not — they are publisher recipes built with --output-tensor-type and --token-embedding-type. Qwen3-8B Q4_K_L is Q4_K_M with token_embd and output at Q8_0, costing 461,885,440 extra bytes (0.43 GiB) for two tensors. No published evidence says that buys anything.

What an importance matrix actually does

An imatrix records the mean squared activation each input channel produced over a calibration corpus. Quantization is a least-squares fit inside each block, and the imatrix supplies the error weights for that fit. From ggml-quants.c, for Q4_K:

weights[l] = qw[l] * sqrtf(sigma2 + x[l]*x[l]);   // with imatrix (qw = importance)
weights[l] = av_x + fabsf(x[l]);                  // without

Without one, llama.cpp weights error by weight magnitude alone; with one, by how much the channel actually mattered. For the K-quants it changes no bit allocation and no file size — an imatrix Q4_K_M and a plain Q4_K_M are byte-for-byte identical in length. It only changes which rounding errors get accepted.

llama.cpp's tools/perplexity/README.md measures this on Llama-3-8B against wikitext-2. Mean KL divergence from FP16 at Q4_K_M: 0.031273 without an imatrix, 0.028152 with — 10% better, free. At Q3_K_S, 0.228217 → 0.199821. At Q2_K, 0.445132 → 0.332223, a 25% gain. It matters more the lower you go, and barely at all above 5 bits.

The same table runs Q2_K with imatrices from 1k, 10k, 100k, 1M and 10M wikitext tokens: KLD 0.337093, 0.331393, 0.331672, 0.335308, 0.332223. The maintainers' summary is "no consistent improvement from using more Wikitext tokens." More calibration data is not the lever people assume.

IQ quants

IQ types store indices into a fixed lattice of quantized vectors rather than independent per-weight values, which is how IQ2_XXS reaches 2.0625 bpw (66 bytes per 256 weights). The low ones hard-require an imatrix — quantize_row_iq2_xxs_impl opens with GGML_ASSERT(quant_weights && "missing quantization weights") — and all of them are slower on CPU, because dequantization becomes a lattice lookup.

They earn it below 4 bits, and two independent datasets agree. Llama-3-8B: iq3_s at 3.42 GiB has KLD 0.111278 while q3_K_S at 3.41 GiB — same size — has 0.199821. Mistral-7B (Artefact2): IQ3_S at 3.52 bpw scores median KLD 0.0205 against Q3_K_S at 3.50 bpw at 0.0304. Between 33% and 44% less divergence at identical size.

Above 4 bits it evaporates: Mistral-7B IQ4_XS at 4.32 bpw scores 0.0088, Q4_K_S at 4.57 bpw scores 0.0083. Same quality, smaller file — fair, but not worth slower CPU inference if you have the room.

unsloth's UD quants

The UD- quants drop the fixed use_more_bits pattern for per-layer, per-model assignment, and the file proves it. In unsloth/Qwen3-8B-GGUF/Qwen3-8B-UD-Q4_K_XL.gguf, layers 0, 1, 6, 16 and 35 get attn_q, attn_k, ffn_gate and ffn_up at Q5_K, while layers 7, 8, 11, 17 and 18 get those same tensors pushed down to IQ4_XS; all 36 attn_v go to Q6_K. Net 5.016 effective bpw against Q4_K_M's 4.911 — 2.1% larger, spent very differently.

Unsloth's Gemma-3-12B comparison puts their 4-bit baseline at KLD 0.024916 / 15.41 GB and the dynamic build at 0.023701 / 15.64 GB: 4.9% better for 1.5% more size, rising to 6% at IQ1_S. Vendor-reported, one model. Evidence the approach works, not a number that transfers.

The part where the data doesn't exist

There is no structured per-model, per-quant quality dataset anywhere. There are four disconnected measurements. Artefact2's gist — the one every bartowski card links to — is Mistral-7B, last updated 2024-02-27. llama.cpp's perplexity README scoreboard is Llama-3-8B, also 2024. Unsloth's tables are Gemma-3, from the party shipping the quants. And one 2026 paper (Kurt, arXiv:2601.14277) covers Llama-3.1-8B-Instruct.

Anyone quoting "Q4_K_M retains 98% of quality" invented it. llama.cpp's quantize help says Q4_K_M costs +0.1754 ppl @ Llama-3-8B; blog posts cite +0.0535 with no model attached.

And there is direct proof these numbers don't transfer. Same README, same harness, same recipe: Llama-2-7B at q2_K has mean KLD 0.108903, Llama-3-8B has 0.445132. Four times the damage on a model of nearly the same size, one generation apart. At q4_K_M it's 0.012686 vs 0.031273, still 2.5x. Llama-3 saw far more tokens per parameter, leaving less redundancy to discard — the direction every model since has moved. When someone says Q4 is fine for the model you just downloaded, they are extrapolating from a 2024 measurement of a different model. Often right; not knowledge.

One shape does generalize. On Llama-3-8B at q4_K_M the median change in the correct token's probability is −0.024%, while the 99th percentile change is 12.084%. Quantization damage is concentrated, not uniform blur — which is why it surfaces on reasoning and code before chat, and why perplexity understates it.

Bigger model at lower quant, or smaller at higher?

The standard metrics cannot settle this. Perplexity is "not directly comparable between models, especially if they use different tokenizers" — llama.cpp's README says so outright. KL divergence is measured against that model's own FP16 logits, so a 70B at IQ2 and an 8B at Q6 are scored against different references. Neither goes on a shared axis.

The best public attempt is llama.cpp discussion #11468: 25 models benchmarked at fixed 3.5 GB, 6.5 GB and 16 GB budgets, putting the crossover at roughly IQ3_XS — above it, spend bytes on parameters; below it, the larger model stops winning. That matches the academic finding that 2-bit large models lose to 4-bit small ones. It is also AlpacaEval win rates, LLM-judged, single-run, no error bars, on Llama-3.1 and Gemma-2 era models. A well-informed prior, not a result.

So: at IQ3_XS and above, more parameters usually beats more bits. Below that, nobody has shown the bigger model is still ahead.

Practical

pip install gguf
gguf-dump model.gguf | head -50                 # see the actual tensor types
llama-quantize --dry-run model-f32.gguf Q4_K    # size without doing the work

llama-imatrix -m model-f16.gguf -f calibration.txt -ngl 99 -o imatrix.gguf
llama-quantize --imatrix imatrix.gguf model-f16.gguf out-q4km.gguf q4_k_m

--pure quantizes everything to one type, which shows what the mixture was buying you. --output-tensor-type q8_0 and --token-embedding-type q8_0 reproduce the _L variants; --tensor-type attn_q=q8_0 targets individual tensors. --allow-requantize exists and you should not use it — requantizing an already-quantized file compounds the error.

MXFP4_MOE is the one honest label: expert tensors go to MXFP4, everything else to Q8_0, and the experts dominate the file.

Then do the boring thing. Download two quants, run your own prompts through both, keep the smaller one if you can't tell them apart. That beats every number on this page, because all of them are about other models.