Best local AI models for 4GB VRAM

Ranked by what actually fits at 32K context, computed from real file bytes.

A 4GB card gives you about 3.72 GiB to work with after driver overhead. 266 indexed models fit at 32K context — the largest being Qwen2.5-Omni-7B at 10.7B parameters in Q2_K.

From the file· fit from summed bytesFrom the file· KV per layer
4GB6GB8GB10GB12GB16GB20GB24GB32GB48GB64GB96GB128GB
alltext generation 266

Fits in 4GB at 32K context

largest quantization that fits, per model
ModelModalityBest quantParams○Total◐Headroom◐
Llama-3.2-1B-Instructtext generationQ8_01.2B3.03 GiB0.69 GiB
ced-basetext generationF3286M1.16 GiB2.56 GiB
gemma-3-1b-ittext generationF161000M2.81 GiB0.91 GiB
embeddinggemma-300mtext generationF32303M2.05 GiB1.67 GiB
Wan2.1-T2V-1.3Btext generationQ5_K_M1.4B3.44 GiB0.28 GiB
LFM2.5-1.2B-Instructtext generationBF161.2B3.37 GiB0.35 GiB
Qwen2.5-1.5B-Instructtext generationQ8_01.5B3.44 GiB0.28 GiB
gemma-3-4b-ittext generationIQ4_XS4.3B3.72 GiB0.00 GiB
MiniCPM5-1B-Claude-Opus-Fable5-Thinkingtext generationF161.1B3.55 GiB0.17 GiB
Qwen2.5-3B-Instructtext generationQ4_K_S3.1B3.65 GiB0.07 GiB
gemma-2-2b-ittext generationIQ2_XS2.6B3.60 GiB0.12 GiB
MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinkingtext generationF161.1B3.55 GiB0.17 GiB
Qwen2.5-0.5B-Instructtext generationF16494M2.08 GiB1.64 GiB
TinyLlama-1.1B-Chat-v1.0text generationF161.1B3.53 GiB0.19 GiB
SmolLM2-135M-Instructtext generationF16135M1.72 GiB2.00 GiB
Qwen2.5-Coder-3B-Instructtext generationQ4_K_S3.1B3.65 GiB0.07 GiB
Qwen2.5-Coder-1.5B-Instructtext generationQ8_01.5B3.44 GiB0.28 GiB
gemma-3-270m-ittext generationF16268M1.38 GiB2.34 GiB
umt5-xxltext generationQ3_K_M5.7B3.70 GiB0.02 GiB
DeepSeek-R1-Distill-Qwen-1.5Btext generationQ8_01.8B3.44 GiB0.28 GiB
LFM2.5-8B-A1BMoEtext generationUD-IQ2_XXS8.5B3.69 GiB0.03 GiB
SmolVLM-500M-Instructtext generationF16507M2.78 GiB0.94 GiB
tinygemma3_cifartext generationQ8_039M0.93 GiB2.79 GiB
VibeThinker-3Btext generationQ4_K_S3.1B3.65 GiB0.07 GiB
Qwen2.5-VL-3B-Instructtext generationQ4_K_S3.8B3.65 GiB0.07 GiB
t5-v1_1-xxltext generationQ4_04.8B3.66 GiB0.06 GiB
Qwen3.6-27B-DFlashtext generationQ8_01.7B2.75 GiB0.97 GiB
LFM2.5-230Mtext generationBF16230M1.57 GiB2.15 GiB
LFM2.5-350Mtext generationBF16354M1.82 GiB1.90 GiB
MiniCPM5-1Btext generationF161.1B3.55 GiB0.17 GiB
TinyLlama-1.1B-Chat-v0.3text generationQ8_01.1B2.57 GiB1.15 GiB
LFM2.5-Audio-1.5Btext generationF161.5B3.52 GiB0.20 GiB
SmolLM2-360M-Instructtext generationF16362M2.69 GiB1.03 GiB
TinyMistral-248M-v2-Instructtext generationQ8_0248M1.40 GiB2.32 GiB
Qwen3.5-DPO-4B-2text generationI1-Q2_K4.2B3.60 GiB0.12 GiB
GLM-OCRtext generationQ8_01.3B3.67 GiB0.05 GiB
Qwen3.6-35B-A3B-DFlashtext generationF16386M1.74 GiB1.98 GiB
Holo-3.1-4Btext generationI1-IQ2_M5.2B3.69 GiB0.03 GiB
Qwen2.5-Coder-0.5B-Instructtext generationF16494M2.08 GiB1.64 GiB
Garnet-OCR-3B-0422text generationI1-IQ4_XS4.1B3.71 GiB0.01 GiB
granite-4.0-h-tinyMoEtext generationQ2_K6.9B3.48 GiB0.24 GiB
NEXUS-Medicaltext generationQ8_01.5B3.21 GiB0.51 GiB
privacy-filter-nemotronMoEtext generationF161.4B3.51 GiB0.21 GiB
DA3-BASEtext generationQ8_0—2.75 GiB0.97 GiB
Qwen2.5-Coder-0.5Btext generationF16494M2.08 GiB1.64 GiB
Qwen2-0.5B-Instructtext generationF32494M3.00 GiB0.72 GiB
MiniCPM5-1B-Agentic-Tooluse-Merged-FP16text generationF161.1B3.55 GiB0.17 GiB
LFM2.5-1.2B-Thinkingtext generationBF161.2B3.37 GiB0.35 GiB
gemma-4-E2B-it-ultra-uncensored-heretictext generationQ3_K_M5.1B3.39 GiB0.33 GiB
Qwen2.5-Coder-3Btext generationQ4_K_S3.1B3.65 GiB0.07 GiB
SmolVLM2-500M-Video-Instructtext generationF16507M2.78 GiB0.94 GiB
privacy-filter-multilingualMoEtext generationF161.4B3.51 GiB0.21 GiB
gemma-4-31B-it-DFlashtext generationQ8_01.5B2.54 GiB1.18 GiB
functiongemma-270m-ittext generationF16268M1.39 GiB2.33 GiB
SmolVLM2-2.2B-Instructtext generationQ8_02.2B2.59 GiB1.13 GiB
Qwen2.5-1.5B-Instruct-uncensoredtext generationQ8_01.8B3.44 GiB0.28 GiB
SmolVLM-256M-Instructtext generationF16256M1.77 GiB1.95 GiB
SmolLM-135M-Instructtext generationF16135M1.72 GiB2.00 GiB
Qwen2.5-Coder-1.5Btext generationQ8_01.5B3.21 GiB0.51 GiB
Qwen2.5-Math-1.5B-Instructtext generationQ8_01.5B3.21 GiB0.51 GiB
○Spec sheet◐Predictedwhat these mean

This page models a generic 4GB accelerator, so it answers what fits rather than how fast it runs. For tokens per second you need a specific card — pick one from hardware, where bandwidth is known.