Apple Silicon for Local AI: What a Mac Actually Buys You

Buy the memory bus, not the chip name

A Mac is worth it when the model does not fit anywhere else. gpt-oss-120b in ggml-org's MXFP4 GGUF is 63,387,346,208 bytes — 59.03 GiB in one file. No consumer NVIDIA card holds that; the RTX 5090 has 32 GB. A 128 GB MacBook Pro holds it with roughly 35 GiB left over for KV cache, inside the default memory budget, in a laptop.

It is a bad deal when you want speed on models that already fit in 24 GB. Token generation tracks memory bandwidth and lands within ~1.5x of a used 3090; prompt processing is compute-bound, and that is where Apple is genuinely behind.

First, though: check the GPU core count, not the marketing name. Apple ships "M4 Max" and "M5 Max" in two bus widths, and the slower one is capped at 36 GB, so you cannot buy your way out with a memory upgrade.

Those are Apple's own figures from the MacBook Pro and Mac Studio spec pages. The 32-core M4 Max generates tokens 33% slower than the 40-core M4 Max, and both are sold as "M4 Max". The 36 GB tier is the cheapest Max and the one you do not want.

The default memory ceiling is not your RAM

macOS does not hand the whole machine to the GPU. Metal reports a recommendedMaxWorkingSetSize, and llama.cpp uses exactly that as its total VRAM budget. Measured on a 24 GB M3 running macOS 26.5.1, it is 19,069,665,280 bytes — 17.76 GiB, or 74.0% of physical memory. Scale that and a 64 GB Mac gives you ~47 GiB, a 128 GB Mac ~94.7 GiB.

Older write-ups quote two-thirds below a 36 GB threshold and three-quarters above it; I measured 74% on a 24 GB machine, so that rule is at best stale. Read the number instead — llama.cpp prints it at startup:

recommendedMaxWorkingSetSize  =  19069.67 MB

You can raise it. The knob is undocumented and resets on reboot:

sysctl iogpu.wired_limit_mb            # 0 means "use the default"
sudo sysctl iogpu.wired_limit_mb=57344 # 56 GiB on a 64 GB Mac

Leave real headroom — 8 GiB is a floor. macOS, the window server, and your editor share that pool, and starving them makes the machine stop responding rather than swap gracefully. Concretely: Llama-3.3-70B-Instruct Q4_K_M is 39.60 GiB, and its KV cache is 0.3125 MiB per token (80 layers, 8 KV heads, head_dim 128, f16), so 32K context adds 10.0 GiB. That is 49.6 GiB against a 47.4 GiB default budget on a 64 GB Mac. It does not fit until you raise the limit or drop to -ctk q8_0 -ctv q8_0, which cuts the KV to 5.31 GiB.

One Metal quirk: llama.cpp keeps its residency sets wired for 180 seconds after the last graph so macOS does not reclaim the weights between requests. Tune it with GGML_METAL_RESIDENCY_KEEP_ALIVE_S if you run an idle server.

Prefill is the real gap, and it moved a lot recently

Same benchmark, same model (Llama-2-7B Q4_0, llama-bench -p 512 -n 128), all on recent builds:

Decode tracks bandwidth closely: the 3090 has 1.52x the M5 Max's bandwidth and generates 1.33x as fast. Prefill does not — the 4090 is 3.6x faster. At those pp512 rates a 32K prompt takes about 9.8 s on the M5 Max versus 2.7 s on a 4090, and pp512 flatters long prompts since attention cost grows with length. If your workflow is agentic coding that re-reads a large context every turn, you will feel this every turn. (Caveat: the NVIDIA rows are from llama.cpp's CUDA scoreboard with flash attention off, the Mac row is a default run with it on auto. Treat the ratios as approximate.)

The more interesting number is the software delta. The same 40-core M5 Max was benchmarked twice — once on the pinned November 2023 commit 8e672ef and once on 52b3df002 from June 2026. Prefill went from 987 to 3347 t/s (3.4x). Decode went from 102.9 to 119.1 t/s (1.16x). Decode was already close to the memory roofline; prefill had been leaving compute on the floor. Part of that 3.4x is M5-specific — llama.cpp now detects Metal tensor support and uses the per-core Neural Accelerators Apple added in M5 — so do not extrapolate it to an M4. But it does mean every published Apple benchmark older than about a year understates prefill.

On the bandwidth side, Apple delivers what it advertises. A synthetic streaming-read Metal kernel on a 100 GB/s M3 peaked at 90.9 GB/s, and real decode on the M5 Max lands at 74% (Q4_0), 78% (F16), and 85% (Q8_0) of its 614 GB/s rating. Unified memory is not a compromise on the bandwidth you were sold.

MLX or llama.cpp

llama.cpp is the safe default. Metal is on by default on macOS, -ngl auto-offloads, GGUF quants appear on Hugging Face within hours of a release, and the KV cache options (-ctk/-ctv q8_0) are the difference between fitting and not fitting on a 64 GB machine.

MLX (pip install mlx-lm) is Apple's own array framework. It genuinely exploits unified memory — arrays live in shared memory with no host/device copies — it supports LoRA and full fine-tuning, and it has distributed inference via mx.distributed. The costs: models must be in MLX format (the mlx-community org on Hugging Face), which lags GGUF for new architectures, and the ecosystem around it is thinner.

Ollama made MLX its Apple silicon path in preview on 30 March 2026, publishing 1154 → 1810 t/s prefill and 58 → 112 t/s decode. Read the footnote: that compares Ollama 0.19 on MLX with NVFP4 weights against Ollama 0.18 on llama.cpp with Q4_K_M weights. Two variables changed, and the post does not name the Mac it ran on. At launch it required more than 32 GB of unified memory and accelerated exactly one model. A real improvement, but not a clean runtime-vs-runtime measurement. Benchmark both yourself; it takes ten minutes and beats anyone's blog post.

Who should and should not buy one

Buy one to run 70B-class dense models or 100B+ MoE models on a machine that fits under a desk, when you generate more than you prefill. A 96 or 128 GB Mac runs gpt-oss-120b at 59.03 GiB natively; matching that on NVIDIA means at least three used 3090s and llama.cpp's sequential layer split, which gives you the capacity of all the cards at roughly one card's token rate. Get the 40-core Max, and more memory than you think you need — it is soldered.

Do not buy one if you already own a 3090 or 4090 and mostly run models under 24 GB; your card is faster at everything. Do not buy one for prefill-heavy long-context agent work, where the 3-4x gap compounds every turn. Do not buy one primarily for image or video generation — diffusion is compute-bound end to end, precisely Apple's weak axis. And do not buy one if you need CUDA, which vLLM, most fine-tuning stacks, and most research code assume.

A 512 GB M3 Ultra holds Qwen3-235B-A22B Q4_K_M (132.39 GiB) with room to spare, and Apple markets it for models "with over 600 billion parameters." That claim is true. Whether it is worth the money is a budget question, not a technical one, and we have no opinion on how you spend yours.