Choosing a GPU for local inference

Two numbers, and neither of them is teraflops

VRAM capacity decides what you can run at all. Memory bandwidth decides how fast it generates. Compute throughput — the TFLOPS number on every spec sheet and in every GPU review — governs prompt processing and almost nothing about the tokens-per-second you actually watch appear.

Generating one token requires reading every weight the model uses, once, from VRAM — a pure streaming operation with essentially no data reuse, so it runs at the speed of memory, not math. Qwen3-32B Q4_K_M from bartowski is 19,762,149,696 bytes. On an RTX 3090 at 936 GB/s, moving that once takes 21.1 ms, capping you at 47.4 tokens/sec no matter how many CUDA cores you have. Fitting llama.cpp's measured behaviour on consumer CUDA — about 79% of theoretical bandwidth achieved plus 0.74 ms fixed per-token overhead — predicts roughly 36 tok/s in practice.

That explains the result everyone eventually rediscovers: a five-year-old 3090 beats a lot of newer cards that crush it in every gaming benchmark.

We earn nothing from any purchase here. No affiliate links, no referral codes, no review samples. That mainly matters because it lets us lead with the recommendation that would cost a commission elsewhere: for many people the right move is to buy nothing.

Your current card is probably fine

Find out what you already have before pricing anything. Real file sizes, so you can check against your own card:

Leave headroom above the weights for the KV cache and you have your answer. An 8 GB card runs an 8B model well; 12 GB covers 14B-class models and gpt-oss-20b at modest context; 16 GB covers most of what people use day to day. If you have 24 GB you are already in the tier this guide would otherwise tell you to buy.

If a model almost fits, try --n-cpu-moe N on an MoE model before considering new hardware — it keeps the expert weights of the first N layers in system RAM while attention stays on the GPU, and because only a few experts are read per token the penalty is far smaller than moving whole layers off. --fit on (now the llama.cpp default) sizes unset parameters to your actual free VRAM.

The tiers, with verified bandwidths

Bandwidths are manufacturer specifications. The last column is the arithmetic ceiling for Qwen3-32B Q4_K_M — bandwidth divided by 19,762,149,696 bytes — not a benchmark; expect roughly 75–80% of it in practice.

Note how compressed the middle is. A 4090 generates about 8% faster than a 3090 despite being a vastly more capable gaming card, because 1008 GB/s is only 8% more than 936 GB/s. And the 5080 has more bandwidth than a 3090 yet still cannot load a 32B model, because 16 GB is 16 GB.

The used 3090, honestly

The 3090 has 24 GB and 936 GB/s, which is the cheapest route to that combination that has ever existed. It is also the last GeForce card with an NVLink connector, which matters if you eventually want two.

What has changed is the price. Through 2024 and much of 2025 these traded around $600–700. Not in July 2026 — the DRAM shortage feeding the AI datacenter buildout has pushed GDDR6 and GDDR7 prices up sharply, and used listings followed. Trackers we checked this session disagree by nearly 2x, quoting roughly $700 to $1,250, so we are not going to print a number and pretend it is stable. Check completed eBay sold listings yourself, and know the direction has been upward for months.

The same shortage puts new cards well above MSRP. The 5090 launched at $1,999 and has not been reliably purchasable there; the RTX PRO 6000 Blackwell went from an $8,565 MSRP to a $13,250 official list price in about sixteen months. Nobody credible is forecasting relief before 2027.

When the 5090 is worth it

When you need more than 24 GB in one card, or when generation speed rather than model quality is your bottleneck. 1792 GB/s is a real 1.9x over a 3090 — the largest generational jump in years — and 32 GB runs a 32B model at long context without compromise. Interactively, ~68 tok/s versus ~36 tok/s is a difference you feel.

It is not worth it for 70B-class models, which 32 GB does not fit either, putting you back into multi-GPU or offload where the bandwidth advantage is diluted. A rumoured 24 GB "5080 Super" has been circulating for months; it remains unannounced and reportedly stalled over 3 GB GDDR7 module costs. Do not plan around it.

Unified memory buys capacity, not speed

Strix Halo (256 GB/s, up to ~96 GB addressable by the GPU) and DGX Spark (273 GB/s, 128 GB) exist to run models that will not fit on a consumer GPU at any price. They are not fast. Be clear-eyed about the scale: DGX Spark's 273 GB/s is exactly an M4 Pro laptop chip's 273 GB/s — half an M4 Max, a third of an M3 Ultra, a seventh of a 5090. On a dense 70B model that is single-digit tokens per second.

The redeeming case is MoE, and it is strong. gpt-oss-120b has 128 experts with 4 active per token, so only about 3.1% of expert bytes are read per token even though all 60.88 GiB must stay resident — resident and per-token are different questions. Community benchmarks on a Beelink GTR9 Pro (Vulkan/RADV) report gpt-oss-120b at roughly 53–56 tok/s on Strix Halo and Qwen3-30B-A3B near 100 tok/s, some 17x what the naive "file size ÷ bandwidth" calculation predicts, on a model no 24 GB card can load at all. These boxes make sense for large MoE models and will disappoint you on dense ones.

Macs sit in between: an M3 Ultra's 819 GB/s is real bandwidth attached to real capacity, and Metal achieves about 93% of theoretical in llama.cpp fits, the best ratio of any backend we have constants for. The catch is price and, since March 2026, availability — Apple removed the 512 GB M3 Ultra option entirely and raised the 256 GB upgrade by $400.

Two GPUs do not give you two GPUs' speed

This is the most common expensive mistake. llama.cpp's default split is -sm layer, and its own help text calls it pipelined: layers and KV are divided across cards, but a token must pass through card 0's layers before card 1's can start. Bandwidth does not aggregate. Two 3090s give you 48 GB of capacity at roughly one 3090's token rate — Llama-3.1-70B Q4_K_M at 39.60 GiB lands near 17 tok/s, essentially what a hypothetical single 48 GB card at 936 GB/s would do.

The alternatives are real but conditional. -sm row splits weights by rows and is genuinely parallel; -sm tensor splits weights and KV and is still marked experimental. Both trade bandwidth gains against per-token synchronisation, so the win depends on model shape and interconnect. vLLM's tensor parallelism does aggregate bandwidth, but pays an all-reduce every layer, and no consumer card since the 3090 has NVLink to carry it — that traffic goes over PCIe.

Buy a second card to run bigger models, not the same model faster. That is a perfectly good reason; just buy it for the right one.

What we would actually do

Check what your current card runs first — the answer surprises people. If you must buy and 24 GB is enough, a used 3090 remains the best capacity-per-dollar despite the price rise, and NVLink leaves the door open. If speed matters most and you can absorb 2026 pricing, the 5090. If you want 100B+ MoE models and accept that dense ones will crawl, a unified-memory box. If none of those is clearly true, waiting costs only time, which is more than can be said for the alternative right now.