Best local AI models for 4GB VRAM
Ranked by what actually fits at 32K context, computed from real file bytes.
A 4GB card gives you about 3.72 GiB to work with after driver overhead. 360 indexed models fit at 32K context — the largest being Qwen2.5-Omni-7B at 10.7B parameters in Q2_K.
From the file· fit from summed bytesFrom the file· KV per layer
allspeech recognition 29vision + language 33text generation 266speech synthesis 17embeddings 13video generation 2
Fits in 4GB at 32K context
largest quantization that fits, per model
| Model | Modality | Best quant | Params○ | Total◐ | Headroom◐ |
|---|---|---|---|---|---|
| nemotron-3.5-asr-streaming-0.6b | speech recognition | F32 | 638M | 3.22 GiB | 0.50 GiB |
| Qwen3.5-4B | vision + language | IQ2_M | 4.7B | 3.63 GiB | 0.09 GiB |
| Qwen3.5-0.8B | vision + language | BF16 | 873M | 2.60 GiB | 1.12 GiB |
| Llama-3.2-1B-Instruct | text generation | Q8_0 | 1.2B | 3.03 GiB | 0.69 GiB |
| gemma-4-E2B-it | vision + language | IQ3_XXS | 5.1B | 3.51 GiB | 0.21 GiB |
| parakeet-tdt-0.6b-v3 | speech recognition | F32 | 627M | 3.18 GiB | 0.54 GiB |
| ced-base | text generation | F32 | 86M | 1.16 GiB | 2.56 GiB |
| Ace-Step1.5 | speech synthesis | F32 | 160M | 2.71 GiB | 1.01 GiB |
| Qwen3-TTS-12Hz-0.6B-Base | speech synthesis | Q4_K | 915M | 1.34 GiB | 2.38 GiB |
| gemma-3-1b-it | text generation | F16 | 1000M | 2.81 GiB | 0.91 GiB |
| whisper-medium | speech recognition | F32 | 764M | 3.69 GiB | 0.03 GiB |
| embeddinggemma-300m | text generation | F32 | 303M | 2.05 GiB | 1.67 GiB |
| Wan2.1-T2V-1.3B | text generation | Q5_K_M | 1.4B | 3.44 GiB | 0.28 GiB |
| LFM2.5-1.2B-Instruct | text generation | BF16 | 1.2B | 3.37 GiB | 0.35 GiB |
| Qwen2.5-1.5B-Instruct | text generation | Q8_0 | 1.5B | 3.44 GiB | 0.28 GiB |
| gemma-3-4b-it | text generation | IQ4_XS | 4.3B | 3.72 GiB | 0.00 GiB |
| whisper-large-v3 | speech recognition | Q8_0 | 1.5B | 2.44 GiB | 1.28 GiB |
| MiniCPM5-1B-Claude-Opus-Fable5-Thinking | text generation | F16 | 1.1B | 3.55 GiB | 0.17 GiB |
| whisper-large-v3-turbo | speech recognition | F16 | 809M | 2.36 GiB | 1.36 GiB |
| Qwen3.5-2B | vision + language | Q8_0 | 2.3B | 3.11 GiB | 0.61 GiB |
| Qwen2.5-3B-Instruct | text generation | Q4_K_S | 3.1B | 3.65 GiB | 0.07 GiB |
| gemma-2-2b-it | text generation | IQ2_XS | 2.6B | 3.60 GiB | 0.12 GiB |
| MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking | text generation | F16 | 1.1B | 3.55 GiB | 0.17 GiB |
| Qwen2.5-0.5B-Instruct | text generation | F16 | 494M | 2.08 GiB | 1.64 GiB |
| embeddinggemma-300m-qat-q8_0-unquantized | embeddings | Q8_0 | 303M | 1.22 GiB | 2.50 GiB |
| Qwen3-ASR-1.7B | speech recognition | Q8_0 | 2.3B | 3.18 GiB | 0.54 GiB |
| TinyLlama-1.1B-Chat-v1.0 | text generation | F16 | 1.1B | 3.53 GiB | 0.19 GiB |
| Qwen3-ASR-0.6B | speech recognition | F16 | 938M | 2.32 GiB | 1.40 GiB |
| SmolLM2-135M-Instruct | text generation | F16 | 135M | 1.72 GiB | 2.00 GiB |
| Qwen2.5-Coder-3B-Instruct | text generation | Q4_K_S | 3.1B | 3.65 GiB | 0.07 GiB |
| KaLM-embedding-multilingual-mini-instruct-v2.5 | embeddings | Q8_0 | 494M | 1.65 GiB | 2.07 GiB |
| nomic-embed-text-v1.5 | embeddings | F32 | 137M | 2.40 GiB | 1.32 GiB |
| Qwen2.5-Coder-1.5B-Instruct | text generation | Q8_0 | 1.5B | 3.44 GiB | 0.28 GiB |
| GigaAM-v3 | speech recognition | F32 | 223M | 1.67 GiB | 2.05 GiB |
| gemma-3-270m-it | text generation | F16 | 268M | 1.38 GiB | 2.34 GiB |
| umt5-xxl | text generation | Q3_K_M | 5.7B | 3.70 GiB | 0.02 GiB |
| DeepSeek-R1-Distill-Qwen-1.5B | text generation | Q8_0 | 1.8B | 3.44 GiB | 0.28 GiB |
| LFM2.5-8B-A1BMoE | text generation | UD-IQ2_XXS | 8.5B | 3.69 GiB | 0.03 GiB |
| jina-embeddings-v5-text-nano | embeddings | F16 | 212M | 2.30 GiB | 1.42 GiB |
| SmolVLM-500M-Instruct | text generation | F16 | 507M | 2.78 GiB | 0.94 GiB |
| tinygemma3_cifar | text generation | Q8_0 | 39M | 0.93 GiB | 2.79 GiB |
| LFM2.5-VL-1.6B | vision + language | BF16 | 1.6B | 3.37 GiB | 0.35 GiB |
| VibeThinker-3B | text generation | Q4_K_S | 3.1B | 3.65 GiB | 0.07 GiB |
| Qwen2.5-VL-3B-Instruct | text generation | Q4_K_S | 3.8B | 3.65 GiB | 0.07 GiB |
| t5-v1_1-xxl | text generation | Q4_0 | 4.8B | 3.66 GiB | 0.06 GiB |
| Qwen3.6-27B-DFlash | text generation | Q8_0 | 1.7B | 2.75 GiB | 0.97 GiB |
| all-MiniLM-L6-v2 | embeddings | F32 | 23M | 1.13 GiB | 2.59 GiB |
| LFM2.5-230M | text generation | BF16 | 230M | 1.57 GiB | 2.15 GiB |
| OmniVoice | speech synthesis | F16 | 613M | 2.37 GiB | 1.35 GiB |
| LFM2.5-350M | text generation | BF16 | 354M | 1.82 GiB | 1.90 GiB |
| parakeet-ctc-0.6b | speech recognition | F32 | 609M | 3.11 GiB | 0.61 GiB |
| whisper-small | speech recognition | F32 | 242M | 1.75 GiB | 1.97 GiB |
| MiniCPM5-1B | text generation | F16 | 1.1B | 3.55 GiB | 0.17 GiB |
| TinyLlama-1.1B-Chat-v0.3 | text generation | Q8_0 | 1.1B | 2.57 GiB | 1.15 GiB |
| whisper-large | speech recognition | Q8_0 | 1.5B | 2.40 GiB | 1.32 GiB |
| VieNeu-TTS-0.3B | speech synthesis | Q8_0 | 244M | 1.94 GiB | 1.78 GiB |
| Wan2.2-TI2V-5B | video generation | Q4_0 | 5.0B | 3.66 GiB | 0.06 GiB |
| neutts-air | speech synthesis | Q8_0 | 748M | 1.90 GiB | 1.82 GiB |
| canary-1b-flash | speech recognition | F16 | 811M | 2.51 GiB | 1.21 GiB |
| LFM2.5-Audio-1.5B | text generation | F16 | 1.5B | 3.52 GiB | 0.20 GiB |
This page models a generic 4GB accelerator, so it answers what fits rather than how fast it runs. For tokens per second you need a specific card — pick one from hardware, where bandwidth is known.