A vision model is two files, and images cost more context than you expect

The main GGUF does not contain the vision tower

A VLM is three parts: a vision tower (a ViT that turns pixels into embeddings), a projector (a small matrix that maps those embeddings into the language model's hidden dimension), and the language model itself. In GGUF, parts one and two are packaged in a separate file — the multimodal projector, or mmproj — and the main model file contains only the language model.

The cleanest proof is a byte comparison. Qwen/Qwen3-VL-8B-Instruct-GGUF ships Qwen3VL-8B-Instruct-Q4_K_M.gguf at 5,027,784,800 bytes. The text-only Qwen/Qwen3-8B-GGUF ships Qwen3-8B-Q4_K_M.gguf at 5,027,783,488 bytes. Those differ by 1,312 bytes — metadata, not weights. The vision half lives entirely in mmproj-Qwen3VL-8B-Instruct-F16.gguf, another 1,159,029,824 bytes.

Hence the usual confusion: you download the model file, it loads perfectly, it answers text prompts perfectly, and nothing warns you anything is missing. Then you send an image and llama-server returns Multimodal data provided, but model does not support multimodal requests. (tools/server/server-common.cpp). You loaded half a model.

# both files, explicitly
llama-server -m Qwen3VL-8B-Instruct-Q4_K_M.gguf \
             --mmproj mmproj-Qwen3VL-8B-Instruct-Q8_0.gguf -c 8192

# or let -hf fetch the mmproj for you
llama-server -hf ggml-org/gemma-3-4b-it-GGUF

-hf downloads the mmproj automatically when the repo has one; --no-mmproj opts out. The projector is offloaded to GPU by default — --no-mmproj-offload keeps it on CPU, trading encode latency for VRAM.

Budget for the mmproj separately, because quantizing harder won't shrink it

(Byte sizes from the HuggingFace blobs API on 2026-07-28, converted to GiB.)

The vision tower is a fixed cost that does not scale with the language model. Gemma 3's mmproj is 34% the size of the 4B's quantized weights and 5% of the 27B's — same tower, wildly different proportion. ggml-org/gemma-3-4b-it-GGUF and ggml-org/gemma-3-27b-it-GGUF publish mmproj-model-f16.gguf at 851,251,104 and 857,739,168 bytes. The entire 6,488,064-byte difference is the projection matrix: 1152 × 5376 × 2 bytes for the 27B minus 1152 × 2560 × 2 for the 4B is exactly 6,488,064. The SigLIP tower is byte-identical between them.

Gemma 3 also publishes its mmproj only in F16 — 0.79 GiB is the floor no matter how hard you squeeze the language model. Qwen and InternVL ship Q8_0 projectors, worth ~0.3-0.5 GiB if you're near the edge.

Image tokens are the cost nobody accounts for

An image does not enter the context as "one image." It enters as a block of tokens that occupy KV cache exactly like text, and there can be thousands of them. How many depends on the tiling strategy, and the two dominant approaches could not be more different:

Fixed grid. Gemma 3 resizes every image to 896×896, cuts it into 14-pixel patches (64×64 = 4096), and pools by a factor of 4 per side — n_merge = 4 in clip.cpp — for exactly 256 tokens per image, always. Note that llama.cpp does not implement Gemma 3's pan-and-scan mode (there is no pan_and_scan anywhere in clip.cpp or mtmd-image.cpp), so a 4K screenshot is squashed into the same 896×896 as a thumbnail.

Dynamic resolution. Qwen-VL scales tokens with pixels. Qwen3-VL uses patch_size: 16 and spatial_merge_size: 2, so one token covers a 32×32 pixel region. That makes the arithmetic trivial:

Qwen3-VL-8B has 36 layers, 8 KV heads and head_dim 128, so f16 KV is 2 × 36 × 8 × 128 × 2 = 147,456 bytes — 144 KiB per token. A single 1024×1024 screenshot costs 144 MiB of KV cache and 1,024 of your context tokens. Five of them and you have spent 5,120 tokens and 720 MiB before the model has read a word of your question.

InternVL3-8B splits the difference with true tiling: 448×448 tiles at 256 tokens each, max_dynamic_patch: 12 plus a thumbnail (use_thumbnail: true), so up to 13 × 256 = 3,328 tokens for one image. Its language model is Qwen2.5-7B — 28 layers, 4 KV heads — so KV is 56 KiB per token, and that worst case is 182 MiB.

The resolution ceiling is lower than the config suggests

Qwen2.5-VL's preprocessor_config.json sets max_pixels: 12845056, which at 28×28 pixels per token is 16,384 tokens for one image. llama.cpp does not honor that. In clip.cpp it calls set_limit_image_tokens(8, 4096) for all Qwen2VL/2.5VL/3VL projectors, and set_limit_image_tokens converts tokens to pixels as patch_size² × n_merge². The practical ceilings:

• Qwen3-VL: 4096 × (16² × 2²) = 4,194,304 pixels, i.e. 2048×2048.
• Qwen2.5-VL: 4096 × (14² × 2²) = 3,211,264 pixels, i.e. 1792×1792.

Anything larger is downscaled to fit — feed it a 4K screenshot expecting 4K detail and you get 2048×2048 worth. Both bounds are adjustable with --image-max-tokens N and --image-min-tokens N.

Going the other way costs accuracy, and llama.cpp says so itself. Load a Qwen-VL model below its threshold and it logs "Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks" and "try adding --image-min-tokens 1024" (llama.cpp issue #16842). If your bounding boxes are garbage, that flag is the first thing to try.

One more thing that bites: enabling multimodal disables context shifting and cache reuse. server-context.cpp logs ctx_shift is not supported by multimodal, it will be disabled and cache_reuse is not supported by multimodal, it will be disabled. A long text chat slides its window when it fills; a vision chat just hits the wall. Size -c for the images you plan to send, not the ones you have sent.

Published parameter counts include the vision tower, and nobody separates them

There is no clean source for "how big is the language model half." Every published count blends the two:

The "language model" column is a subtraction against the matching text-only checkpoint (Qwen3-8B, Qwen2.5-7B-Instruct), not a published figure — and it only works because those particular VLMs are built on released text models. Gemma 3 has no text-only 4B twin, so there is nothing to subtract from; the ~0.42B is inferred from the mmproj file size, not read from a manifest. As a sanity check, the 1,159,029,824-byte Qwen3-VL projector against 576,388,336 parameters is 16.09 bits each — F16 plus a few F32 tensors, so the subtraction lands where it should.

A model marketed as "8B" is 8.77B on disk, and roughly 0.58B of that never does language modeling. Treat the headline number as the pipeline total.

Why there are no VLM quality benchmarks on this page

The OpenVLM Leaderboard is the main public cross-model comparison, and its data file (opencompass.openxlab.space/assets/OpenVLM.json) carries the timestamp 20250917132916 — 17 September 2025, over ten months ago. It has 285 models, including Qwen2.5-VL-7B, InternVL3-8B and Gemma3-4B/12B/27B. It contains zero Qwen3-VL entries — an entire flagship family released after the snapshot froze. The Space wrapping it was last touched 2026-03-11, and only to bump Gradio and hub versions.

It also illustrates the parameter problem neatly. It lists Qwen2.5-VL-7B at 8.29B with Language Model Qwen2.5-7B and Vision Model QwenViT — it knows there are two components, names both, and still reports one blended number. Of its 220 open-source entries, 212 name a vision model and none give its parameter count.

We can tell you how many bytes each file is and how many tokens an image will consume, because those are read from the files and the preprocessing source. Ranking VLM quality against a ten-month-old snapshot that omits the models you are most likely choosing between would be worse than saying nothing. Run your own evaluation on your own images.