Running a model bigger than your VRAM

Start by passing no placement flags at all

If the model is a mixture-of-experts, you can run one several times the size of your VRAM at usable speed. If it is dense, you mostly cannot. That is the entire answer, and the reason is that MoE weights are resident in bulk but read sparsely.

So the first thing to try is nothing: llama-server -m gpt-oss-120b-MXFP4.gguf. Recent llama.cpp defaults to -ngl auto with --fit on — in common/common.h, n_gpu_layers = -1 and fit_params = true — and the pass in common/fit.cpp measures free device memory, then decides how many layers' expert tensors to leave in system RAM. The old question, "how many layers do I offload," is now llama.cpp's problem. Yours is what context length and what quant you can afford.

Resident is not the same as read

ggml-org's gpt-oss-120b-MXFP4.gguf is 63,387,346,208 bytes on disk, of which 63,374,323,968 bytes (59.02 GiB) is tensor data. Parsing the header: 687 tensors, 36 blocks, and the split is lopsided.

The expert tensors — ffn_up_exps, ffn_gate_exps, ffn_down_exps and their biases — account for 61,073,326,080 bytes, or 96.4% of the model. Everything else, meaning all attention weights, all norms, the routers, the embedding and the output projection, comes to 2,300,997,888 bytes: 2.14 GiB. That fits on a 6GB card with room to spare.

The header also says expert_count = 128 and expert_used_count = 4. Four of 128 is 3.125%, so per token the routed expert weights actually read are 61,073,326,080 / 32 = 1,908,541,440 bytes — 1.78 GiB out of 56.88 GiB resident. Per layer, that is 53,015,040 bytes read of 1,696,481,280 bytes held.

All 128 experts have to be somewhere addressable, because the router picks a different four for every token and you cannot predict which. But whichever memory they sit in only has to deliver about a thirty-second of them per token. Slow memory is a fine place to put bytes you rarely read.

Why this does not work for dense models

Contrast bartowski's Llama-3.3-70B-Instruct-Q4_K_M.gguf: 42,512,531,712 bytes of tensor data across 80 blocks, averaging 513,245,184 bytes per layer, and there is no sparsity anywhere. Every byte of every layer you push to system RAM is read in full for every token.

Put both on a 24GB card with 64GB of system RAM:

The larger model moves sixteen times less data across the slow bus. This is the whole reason MoE is the shape that offloads: the tensors that are too big to fit are exactly the tensors you barely touch.

(96 GB/s is dual-channel DDR5-6000 at its theoretical rate. Nothing achieves that. Treat both floors as ceilings on speed you will not reach — see the caveat below.)

The flags, and what they now do

-ncmoe N / --n-cpu-moe N keeps the expert tensors of the first N layers in system memory. In common/arg.cpp it expands to one buffer-type override per layer matching blk\.N\.ffn_(up|down|gate|gate_up)_(ch|)exps. -cmoe / --cpu-moe does all layers. Both leave attention, norms and routers on the GPU, which is the point — those are the tensors that are read in full every token.

One thing to know before you reach for them: setting -ngl, -ncmoe, -cmoe or -ot yourself turns off the automatic placement. fit.cpp throws ("n_gpu_layers already set by user … abort", "model_params::tensor_buft_overrides already set by user, abort"), logs a warning, and skips the layer-allocation step. It happens to check these after it has already trimmed the context, so you may still get a reduced -c and no expert placement. Override deliberately or not at all.

What --fit does when left alone, in order: if you did not pass -c, it shrinks the context toward fit_params_min_ctx (4096) to claw back memory; then it fills devices back-to-front with dense-only layers, keeping every layer's attention on the GPU and its experts on the CPU; then, for MoE models, it converts dense-only layers back into full layers front-to-back until the card is full, down to fractions of a single layer. It aims to leave 1 GiB free per device (fit_params_target). Build llama-fit-params and run it with -fitp on to see the estimate without loading.

Working it out for a 24GB card

Take a 3090 with 64GB of system RAM, gpt-oss-120b, 32K context.

KV cache first, because it comes out of the same budget. The GGUF gives head_count_kv = 8, key_length = 64, value_length = 64: at f16 that is 2048 bytes per token per layer. gpt-oss alternates sliding and full attention, 18 layers each, with sliding_window = 128, and llama.cpp's CLI tools default swa_full = false, so llama-kv-cache-iswa.cpp sizes the sliding cache as GGML_PAD(min(n_ctx, n_swa + n_ubatch), 256) = 768 cells. So 18 × 32768 × 2048 plus 18 × 768 × 2048 = 1.15 GiB, not the 2.25 GiB a flat 36-layer formula would give.

Now the budget. Roughly 23.5 GiB free, less llama.cpp's 1 GiB margin, less 2.14 GiB of dense weights, less 1.15 GiB of KV, less compute buffers (a few hundred MiB — llama.cpp measures this, we won't guess). That leaves about 18.5 GiB for expert tensors, and at 1.58 GiB per layer that is eleven or twelve layers' worth. The other 25 layers keep their experts in system RAM: 39.5 GiB, comfortable inside 64GB with the OS and page cache. -ncmoe 25 is the manual equivalent.

Per token the GPU then reads about 2.27 GB (dense weights minus the embedding table, which is a single row lookup, plus eleven layers of routed experts) and the CPU reads 1.33 GB. Using the fitted consumer-CUDA constants — 79% of the 936 GB/s a 3090 advertises, plus 0.74 ms fixed — the GPU leg is about 3.8 ms. The CPU leg at DDR5-6000's theoretical 96 GB/s is 13.8 ms. They are serial: layer i's expert FFN must finish on the CPU before layer i+1's attention runs on the GPU. So roughly 17.6 ms per token, or 57 tok/s.

Do not believe that number. We have fitted efficiency constants for Metal and consumer CUDA; we have not fitted one for CPU expert matmuls, and real DRAM efficiency on gather-heavy reads is well under 100%. If the CPU leg lands at 60–70% of theoretical, you are at 20–23 ms on that side and 35–45 tok/s overall. Run llama-bench on your own box rather than trusting anyone's arithmetic, including this page's.

PCIe is not the bottleneck

This is the most repeated wrong claim about offload. Weights are not streamed across PCIe per token. They are loaded once into whichever buffer they live in and stay there. What crosses the bus is the activation at the layer boundary.

For gpt-oss, that is the hidden state: 2880 floats, F32, 11,520 bytes, going GPU→CPU before the expert FFN and CPU→GPU after. Twenty-five offloaded layers means 25 × 23,040 = 576,000 bytes per token — about 562 KiB. On PCIe 4.0 x16 (31.5 GB/s theoretical, ~25 GB/s measured) that is 23 microseconds. On a wretched PCIe 3.0 x4 riser it is still under 200 microseconds.

Against a 17.6 ms token, PCIe is a tenth of a percent. The DDR bus carries 1.33 GB per token while PCIe carries 0.00058 GB — a ratio of about 2,300 to 1. Two serial rooflines, GPU bandwidth and system-RAM bandwidth, decide your speed. The link between them does not. Buying a motherboard for its PCIe lanes to fix offload performance is buying the wrong thing; more memory channels is the purchase that would actually move the number, and even that is often not worth it.

What actually degrades

Prefill. Token generation is bandwidth-bound and survives offload because of the sparsity above; prompt processing is compute-bound, and for the offloaded layers that compute now happens on your CPU cores. A long system prompt or a large pasted file will feel far worse than generation does. If you are pasting big contexts, budget for that and enable prompt caching in llama-server rather than assuming something is broken.

The other failure mode is exceeding physical RAM. llama.cpp mmaps the GGUF, so a model that nearly fits will run off the page cache and feel fine; a model that does not fit will thrash your SSD and drop to seconds per token. 39.5 GiB of experts in 64GB is fine. The same setup with 32GB is not, and no flag will save it — that is the case where a smaller MoE like Qwen3-30B-A3B (18.55 GB total, 94.6% of it experts, 8 of 128 active) is the honest recommendation instead.