Best local AI models for 8GB VRAM
Ranked by what actually fits at 32K context, computed from real file bytes.
A 8GB card gives you about 7.44 GiB to work with after driver overhead. 986 indexed models fit at 32K context — the largest being Darwin-36B-Opus at 34.7B parameters in IQ2_M.
From the file· fit from summed bytesFrom the file· KV per layer
allvision + language 88speech recognition 38text generation 806speech synthesis 20embeddings 25video generation 8image generation 1
Fits in 8GB at 32K context
largest quantization that fits, per model
| Model | Modality | Best quant | Params○ | Total◐ | Headroom◐ |
|---|---|---|---|---|---|
| Qwen3.5-9B | vision + language | Q4_K_M | 9.7B | 7.30 GiB | 0.14 GiB |
| gemma-4-12B-it | vision + language | UD-IQ2_M | 12.0B | 7.24 GiB | 0.20 GiB |
| nemotron-3.5-asr-streaming-0.6b | speech recognition | F32 | 638M | 3.22 GiB | 4.22 GiB |
| Qwen3.5-4B | vision + language | Q8_0 | 4.7B | 6.12 GiB | 1.32 GiB |
| Qwythos-9B-Claude-Mythos-5-1M | vision + language | Q4_K | 9.4B | 7.22 GiB | 0.22 GiB |
| gemma-4-E4B-it | text generation | Q5_K_M | 8.0B | 6.69 GiB | 0.75 GiB |
| Qwen3-4B | text generation | IQ4_XS | 4.0B | 7.43 GiB | 0.01 GiB |
| Qwen3.5-0.8B | vision + language | BF16 | 873M | 2.60 GiB | 4.84 GiB |
| Llama-3.2-1B-Instruct | text generation | F16 | 1.2B | 4.11 GiB | 3.33 GiB |
| gemma-4-E2B-it | vision + language | Q8_0 | 5.1B | 5.74 GiB | 1.70 GiB |
| gemma-4-E4B-it-qat-q4_0-unquantized | vision + language | Q4_0 | 7.9B | 6.12 GiB | 1.32 GiB |
| parakeet-tdt-0.6b-v3 | speech recognition | F32 | 627M | 3.18 GiB | 4.26 GiB |
| Llama-3.1-8B-Instruct | text generation | UD-IQ2_XXS | 8.0B | 7.17 GiB | 0.27 GiB |
| ced-base | text generation | F32 | 86M | 1.16 GiB | 6.28 GiB |
| Ace-Step1.5 | speech synthesis | F32 | 160M | 2.71 GiB | 4.73 GiB |
| Qwen2.5-7B-Instruct | text generation | Q4_K_L | 7.6B | 7.34 GiB | 0.10 GiB |
| Qwen3-TTS-12Hz-0.6B-Base | speech synthesis | Q4_K_M | 915M | 5.57 GiB | 1.87 GiB |
| Qwythos-9B-v2 | vision + language | I1-Q4_1 | 9.7B | 7.39 GiB | 0.05 GiB |
| UI-TARS-1.5-7B | text generation | Q4_K_M | 8.3B | 6.97 GiB | 0.47 GiB |
| gemma-3-1b-it | text generation | F16 | 1000M | 2.81 GiB | 4.63 GiB |
| gemma-4-E2B-it-qat-q4_0-unquantized | text generation | Q4_K | 5.1B | 4.22 GiB | 3.22 GiB |
| Qwen3-1.7B | text generation | Q8_0 | 2.0B | 6.31 GiB | 1.13 GiB |
| whisper-medium | speech recognition | F32 | 764M | 3.69 GiB | 3.75 GiB |
| embeddinggemma-300m | text generation | F32 | 303M | 2.05 GiB | 5.39 GiB |
| Llama-3.2-3B-Instruct | text generation | Q6_K_L | 3.2B | 6.86 GiB | 0.58 GiB |
| Qwen3-0.6B | text generation | BF16 | 752M | 5.68 GiB | 1.76 GiB |
| Voxtral-Mini-4B-Realtime-2602 | speech recognition | Q5_K_M | 4.4B | 7.12 GiB | 0.32 GiB |
| Wan2.1-T2V-1.3B | text generation | Q4_0 | 1.4B | 7.32 GiB | 0.12 GiB |
| LFM2.5-1.2B-Instruct | text generation | BF16 | 1.2B | 3.37 GiB | 4.07 GiB |
| Qwen2.5-Coder-7B-Instruct | text generation | Q4_K_L | 7.6B | 7.34 GiB | 0.10 GiB |
| Qwen3-VL-4B-Instruct | vision + language | IQ4_XS | 4.4B | 7.43 GiB | 0.01 GiB |
| Qwen2.5-1.5B-Instruct | text generation | F16 | 1.5B | 4.56 GiB | 2.88 GiB |
| Jan-v3-4B-base-instruct | text generation | Q3_K_M | 4.4B | 7.40 GiB | 0.04 GiB |
| gemma-3-4b-it | text generation | Q8_0 | 4.3B | 5.46 GiB | 1.98 GiB |
| whisper-large-v3 | speech recognition | F16 | 1.5B | 3.74 GiB | 3.70 GiB |
| MiniCPM5-1B-Claude-Opus-Fable5-Thinking | text generation | F16 | 1.1B | 3.55 GiB | 3.89 GiB |
| Qwen2.5-VL-7B-Instruct | vision + language | Q4_K_L | 8.3B | 7.34 GiB | 0.10 GiB |
| Qwen3-VL-2B-Instruct | vision + language | Q8_0 | 2.1B | 6.00 GiB | 1.44 GiB |
| Ornith-1.0-9B | text generation | Q4_1 | 9.2B | 7.37 GiB | 0.07 GiB |
| jina-embeddings-v5-text-small | embeddings | F16 | 596M | 5.39 GiB | 2.05 GiB |
| whisper-large-v3-turbo | speech recognition | F16 | 809M | 2.36 GiB | 5.08 GiB |
| Qwen3-4B-Instruct-2507 | text generation | IQ4_XS | 4.0B | 7.43 GiB | 0.01 GiB |
| granite-4.1-3b | text generation | Q8_0 | 3.4B | 6.67 GiB | 0.77 GiB |
| gemma-3-12b-it | vision + language | UD-IQ2_M | 12.2B | 7.38 GiB | 0.06 GiB |
| Qwen3.5-2B | vision + language | BF16 | 2.3B | 4.80 GiB | 2.64 GiB |
| Qwen2.5-3B-Instruct | text generation | Q8_0 | 3.1B | 5.30 GiB | 2.14 GiB |
| gemma-2-2b-it | text generation | Q8_0 | 2.6B | 5.26 GiB | 2.18 GiB |
| MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking | text generation | F16 | 1.1B | 3.55 GiB | 3.89 GiB |
| Qwen2.5-0.5B-Instruct | text generation | F16 | 494M | 2.08 GiB | 5.36 GiB |
| embeddinggemma-300m-qat-q8_0-unquantized | embeddings | Q8_0 | 303M | 1.22 GiB | 6.22 GiB |
| Qwen3-ASR-1.7B | speech recognition | F16 | 2.3B | 5.23 GiB | 2.21 GiB |
| gemma-4-12b-heretic-abliterated | text generation | I1-IQ2_M | 12.0B | 7.39 GiB | 0.05 GiB |
| TinyLlama-1.1B-Chat-v1.0 | text generation | F16 | 1.1B | 3.53 GiB | 3.91 GiB |
| Phi-4-mini-instruct | text generation | Q5_K_S | 3.8B | 7.35 GiB | 0.09 GiB |
| Qwen3-ASR-0.6B | speech recognition | F16 | 938M | 2.32 GiB | 5.12 GiB |
| SmolLM2-135M-Instruct | text generation | F16 | 135M | 1.72 GiB | 5.72 GiB |
| Wan2.2-Animate-14B | video generation | Q2_K | 17.3B | 7.20 GiB | 0.24 GiB |
| Qwen3.5-9B | vision + language | Q4_K_M | 9.7B | 7.22 GiB | 0.22 GiB |
| FastContext-1.0-4B-SFT | text generation | I1-IQ4_XS | 4.0B | 7.43 GiB | 0.01 GiB |
| DeepSeek-R1-Distill-Qwen-7B | text generation | Q4_K_L | 7.6B | 7.34 GiB | 0.10 GiB |
This page models a generic 8GB accelerator, so it answers what fits rather than how fast it runs. For tokens per second you need a specific card — pick one from hardware, where bandwidth is known.