llama.cpp
The engine most other tools wrap. Widest format and hardware support, and where new architectures land first.
From the file· capabilities, not benchmarks
What it is
A C++ inference engine with no dependencies to speak of, running GGUF files across CUDA, Metal, ROCm, Vulkan, SYCL and plain CPU. Almost every friendlier tool in local AI is a wrapper around it, so its capabilities set the ceiling for the others.
Capabilities
| Model formats | GGUF |
| KV cache quantization | yes |
| CPU offload | yes |
| MoE expert offload | yes |
| Multi-GPU | Layer split by default — capacity adds up, bandwidth does not. Row split available. |
| Concurrency | Single-user focused. A server exists but is not built for heavy concurrency. |
| Platforms | Linux, macOS, Windows |
Best for
Anyone who wants the newest architectures, the most quantization choices, or the most control over memory.
Watch out for
It is a command-line tool with a lot of flags. The defaults have improved considerably — it now sizes offload automatically — but it expects you to know what you are asking for.