# ossmodeldb — exact memory and modeled speed for models you run yourself.
# Weights are summed from published file bytes, never estimated from a constant.
# KV cache is computed per layer, including sliding-window, MLA and hybrid architectures.
# Data: CC BY 4.0. 2382 base models, 67146 quantizations, 239 accelerators.

## Machine interface
- https://ossmodeldb.org/api/v1 — self-describing discovery document, keyless, CORS *
- https://ossmodeldb.org/api/v1/models?modality=&limit= — base models with exact size ranges
- https://ossmodeldb.org/api/v1/fit?model={slug}&hw={slug}&ctx={n}&kv={f16|q8_0|q4_0} — fit a model to hardware
- https://ossmodeldb.org/api/gguf?url={hf-gguf-url} — parse any Hub GGUF header live over HTTP Range

## Method, in one line each
- A quantization label names a MIXTURE, not a uniform precision. Effective bits per weight is always
  above the nominal rate because attention and output tensors are kept higher.
- KV cache is a per-layer sum. Three families break the flat context x layers x heads formula:
  sliding-window attention caches a fixed window on most layers; MLA allocates no V cache at all;
  hybrid linear-attention layers keep a fixed-size state that does not grow with context.
- Token generation is bound by memory bandwidth, not teraflops. Capacity decides what runs;
  bandwidth decides how fast.
- Mixture-of-experts models must hold every expert resident but read only the routed ones per token.
  Resident size and per-token bytes are different questions and should never be collapsed.

## Exact facts
- Qwen3.6-35B-A3B Q4_K_M is exactly 21166757728 bytes (19.713 GiB), an effective 4.7100 bits per weight.
- Qwen3.5-9B Q4_K_M is exactly 5680522464 bytes (5.290 GiB), an effective 4.7077 bits per weight.
- Qwen3.5-9B Q4_K_M is exactly 5868826976 bytes (5.466 GiB), an effective 4.8638 bits per weight.
- Qwen3.5-9B Q4_K_M is exactly 5627044256 bytes (5.241 GiB), an effective 4.6634 bits per weight.
- gemma-4-26B-A4B-it Q4_K_M is exactly 17035039872 bytes (15.865 GiB), an effective 5.1341 bits per weight.
- nemotron-3.5-asr-streaming-0.6b Q4_K_M is exactly 495831520 bytes (0.462 GiB), an effective 6.2174 bits per weight.
- Qwen3-Coder-30B-A3B-Instruct Q4_K_M is exactly 18556689760 bytes (17.282 GiB), an effective 4.8622 bits per weight.
- Qwen3-Coder-30B-A3B-Instruct Q4_K_M is exactly 18556689568 bytes (17.282 GiB), an effective 4.8622 bits per weight.
- Qwen3-Coder-30B-A3B-Instruct Q4_K_M is exactly 18632186176 bytes (17.353 GiB), an effective 4.8820 bits per weight.
- parakeet-unified-en-0.6b Q4_K_M is exactly 477274496 bytes (0.444 GiB), an effective 6.1752 bits per weight.
- Qwen3.5-4B Q4_K_M is exactly 2740937888 bytes (2.553 GiB), an effective 4.7056 bits per weight.
- Qwen3.5-4B Q4_K_M is exactly 2707513696 bytes (2.522 GiB), an effective 4.6482 bits per weight.

## Guides
- https://ossmodeldb.org/guides/multi-gpu-for-local-ai — llama.cpp's default layer split runs your GPUs one at a time, so two 3090s give you 48GB at roughly one card's token rate. Here is the arithmetic.
- https://ossmodeldb.org/guides/vision-language-models-locally — VLMs need a separate mmproj file alongside the main GGUF, and one image can burn 4096 context tokens — 576 MiB of KV cache on Qwen3-VL-8B.
- https://ossmodeldb.org/guides/apple-silicon-for-local-ai — Unified memory buys capacity no consumer GPU can match, but prefill is 3-4x slower and the same chip name ships at two bandwidths 33% apart.
- https://ossmodeldb.org/guides/choosing-a-quantization — Q4_K_M is a good default and Q5_K_M or Q6_K if it fits — but the per-model data needed to do better than that is not published anywhere.
- https://ossmodeldb.org/guides/choosing-a-gpu — Capacity decides what you can run; memory bandwidth decides how fast. Teraflops barely matter for generation, and your current card may already be enough.
- https://ossmodeldb.org/guides/choosing-a-runtime — Most of these are layers over llama.cpp, not rivals to it. Pick a wrapper for ergonomics; switch to vLLM only when you serve concurrent requests.
- https://ossmodeldb.org/guides/what-quantization-costs-you — On the one model with public measurements, Q4_K_M costs 2.4% perplexity. The step from Q4 to Q3 costs more than everything above it combined.
- https://ossmodeldb.org/guides/local-embeddings-and-rag — Retrieval models run from 91 MiB to 1.1 GiB and work fine on CPU. Your real costs are vector storage, the reranker pass, and MTEB's doubled memory field.
- https://ossmodeldb.org/guides/moe-and-cpu-offload — MoE models offload to system RAM well because all experts must be resident but only a few are read per token. Here is the arithmetic and the llama.cpp flags.
- https://ossmodeldb.org/guides/local-speech-tts-and-asr — Voice models are 60 MB to 3 GB, not 30 GB. Almost any laptop runs them, but the RTF figures you've read were measured on an H200 at batch 64.
- https://ossmodeldb.org/guides/running-your-first-model — Pick a runtime, pick a GGUF file that actually fits your VRAM, and know how to tell whether it fit.
- https://ossmodeldb.org/guides/when-it-does-not-fit — Hard OOM, mid-context OOM, and the silent spill that runs 5-20x slow are different problems with different fixes. Diagnose first, then work down the ladder.
- https://ossmodeldb.org/guides/local-image-and-video-generation — A diffusion model is a pipeline of four models, not one file, and the headline parameter count covers only the denoiser. Here is the real memory math.
- https://ossmodeldb.org/guides/kv-cache-and-context — KV cache is sized by num_key_value_heads and allocated up front. For Gemma-3, DeepSeek, and Qwen3.5/3.6 the flat formula overstates it 4x to 71x.

## Caveats we hold ourselves to
- Speed figures on this site are MODELED, not measured, and carry an error band. Nothing is labelled
  measured until someone has run it.
- Image, video and speech models have exact memory but no throughput data: no public source has measured
  it on consumer hardware, and we decline to invent it.
- Where a model's sliding-window layout cannot be resolved, we show no KV total rather than a wrong one.

## Method and corrections
- https://ossmodeldb.org/methodology
- https://ossmodeldb.org/corrections
- https://ossmodeldb.org/status