What AI models can a A100 40GB run?

2032 of 2118 indexed open-weight models fit a A100 40GB at 4,096 context with q4_0 KV cache, the largest being deepseek-llm-67b-chat at I1-Q4_0. That covers text, vision-language, image, video and speech models.

How much usable memory does a A100 40GB actually have?

Its nameplate is 40 GB, but about 37.20 GiB is available to a model once driver and compositor overhead is accounted for.

Is a A100 40GB fast for local AI?

Its memory bandwidth is 1555 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.

NVIDIA · datacenter

A100 40GB

A100 40GB has 40 GB of VRAM at 1555 GB/s — about 37.20 GiB usable after driver and compositor overhead. 2032 of 2118 indexed models fit at 4K context with q4_0 KV.

Spec sheet· bandwidth, theoreticalFrom the file· fit from summed bytesPredicted· speed

Memory

40 GB

HBM2

Bandwidth

1555 GB/s

5120-bit bus

Tensor FP16

312 TF

dense

TDP

400 W

Context4K 8K 16K 32K 64K 128K

KV cachef16 q8_0 q4_0quantizing the KV cache is a ~2× lever on the dominant term at long context

text 1745vision language 183video 16audio tts 21image 2embedding 26audio asr 39

What fits at 4K context

largest quantization that fits, per model · 2032 of 2118 indexed

Model	Best quant	Params	Weights●	KV●	Total in memory◐	Headroom◐	tok/s◐
deepseek-llm-67b-chat	I1-Q4_0	67.4B	35.65 GiB	0.42 GiB	37.16 GiB	0.04 GiB	25±22%
deepseek-llm-67b-base	I1-Q4_0	67.4B	35.65 GiB	0.42 GiB	37.16 GiB	0.04 GiB	25±22%
openbuddy-deepseek-67b-v15.3-4k	I1-Q4_0	67.4B	35.65 GiB	0.42 GiB	37.16 GiB	0.04 GiB	25±22%
Seed-OSS-36B-Instruct	Q8_0	36.2B	35.78 GiB	0.28 GiB	37.16 GiB	0.04 GiB	25±22%
Hermes-4.3-36B-heretic	Q8_0	36.2B	35.78 GiB	0.28 GiB	37.16 GiB	0.04 GiB	25±22%
Hermes-4.3-36B	Q8_0	36.2B	35.78 GiB	0.28 GiB	37.16 GiB	0.04 GiB	25±22%
Seed-OSS-36B-Base	Q8_0	36.2B	35.78 GiB	0.28 GiB	37.16 GiB	0.04 GiB	25±22%
GLM-4.6VMoE	UD-IQ1_M	108B	35.93 GiB	0.20 GiB	37.16 GiB	0.04 GiB	107±37%
Qwen3.5-88BMoE	I1-IQ3_M	87.7B	36.10 GiB	0.03 GiB	37.15 GiB	0.05 GiB	128±37%
Mistral-Small-Instruct-2409	IQ3_XS	22.2B	35.84 GiB	0.25 GiB	37.15 GiB	0.05 GiB	25±22%
command-r-35b-writer-v2	Q8_0	35.0B	34.63 GiB	1.41 GiB	37.14 GiB	0.06 GiB	25±22%
Maenad-70B	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B-Uncensored-v2-Unbiased-Reasoner	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
Strawberrylemonade-L3-70B-v1.2	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
Llama-3.3_70_b_uncensored_continued	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
Llama-3.3-70B-Instruct-abliterated	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
grok-oss-Revenant-70B	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
Hermes-4-70B-heretic	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
Golem-70B-v1b	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B-heretic	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
Infinity-Instruct-7M-Gen-Llama3_1-70B	IQ4_XS	70.6B	35.64 GiB	0.35 GiB	37.12 GiB	0.08 GiB	25±22%
GLM-4.5-AirMoE	UD-IQ1_S	110B	35.86 GiB	0.20 GiB	37.09 GiB	0.11 GiB	107±37%
Llama-3_1-Nemotron-51B-Instruct	Q5_K_S	51.5B	33.12 GiB	2.81 GiB	37.07 GiB	0.13 GiB	25±22%
Qwen3.5-122B-A10BMoE	UD-IQ1_M	125B	36.02 GiB	0.03 GiB	37.07 GiB	0.13 GiB	144±37%
Hunyuan-A13B-InstructMoE	Q3_K_M	80.4B	35.91 GiB	0.14 GiB	37.05 GiB	0.15 GiB	25±22%
xLAM-8x7b-rMoE	Q6_K_L	46.7B	35.80 GiB	0.14 GiB	36.98 GiB	0.22 GiB	46±37%
Qwen3.5-99BMoE	I1-IQ3_XXS	99.0B	35.87 GiB	0.03 GiB	36.93 GiB	0.27 GiB	135±37%
Hypernova-60B-2605MoE	I1-Q4_K_S	58.7B	35.89 GiB	0.04 GiB	36.92 GiB	0.28 GiB	122±37%
dolphin-2.6-mixtral-8x7bMoE	I1-Q6_K	46.7B	35.74 GiB	0.14 GiB	36.92 GiB	0.28 GiB	47±37%
Nous-Hermes-2-Mixtral-8x7B-DPOMoE	Q6_K	46.7B	35.74 GiB	0.14 GiB	36.92 GiB	0.28 GiB	47±37%
Mixtral-8x7B-Instruct-v0.1MoE	Q6_K	46.7B	35.74 GiB	0.14 GiB	36.92 GiB	0.28 GiB	47±37%
Open_Gpt4_8x7B_v0.1MoE	Q6_K	46.7B	35.74 GiB	0.14 GiB	36.92 GiB	0.28 GiB	47±37%
dolphin-2.5-mixtral-8x7bMoE	Q6_K	46.7B	35.74 GiB	0.14 GiB	36.92 GiB	0.28 GiB	47±37%
dolphin-2.7-mixtral-8x7bMoE	Q6_K	46.7B	35.74 GiB	0.14 GiB	36.92 GiB	0.28 GiB	47±37%
Mixtral-8x7B-v0.1MoE	Q6_K	46.7B	35.74 GiB	0.14 GiB	36.92 GiB	0.28 GiB	47±37%
Mixtral-8x7B-MoE-RP-StoryMoE	Q6_K	46.7B	35.74 GiB	0.14 GiB	36.92 GiB	0.28 GiB	47±37%
Open_Gpt4_8x7B_v0.2MoE	Q6_K	46.7B	35.74 GiB	0.14 GiB	36.92 GiB	0.28 GiB	47±37%
Llama-3_3-Nemotron-Super-49B-v1_5	Q5_K_M	49.9B	32.96 GiB	2.81 GiB	36.91 GiB	0.29 GiB	25±22%
Valkyrie-49B-v2.1	I1-Q5_K_M	49.9B	32.96 GiB	2.81 GiB	36.91 GiB	0.29 GiB	25±22%
Llama-3_3-Nemotron-Super-49B-v1	Q5_K_M	49.9B	32.96 GiB	2.81 GiB	36.91 GiB	0.29 GiB	25±22%
Huihui-GLM-4.7-Flash-abliterated-57BMoE	I1-Q5_K_S	57.3B	35.72 GiB	0.15 GiB	36.91 GiB	0.29 GiB	109±37%
Apertus-70B-Instruct-2509	IQ4_XS	70.6B	35.36 GiB	0.35 GiB	36.89 GiB	0.31 GiB	25±22%
Qwen3-Coder-NextMoE	UD-IQ4_XS	79.7B	35.79 GiB	0.11 GiB	36.89 GiB	0.31 GiB	159±37%
GLM-4.5VMoE	I1-IQ2_XXS	108B	35.61 GiB	0.20 GiB	36.84 GiB	0.36 GiB	108±37%
CalmeRys-78B-Orpo-v0.1	I1-IQ3_M	78.0B	35.33 GiB	0.38 GiB	36.83 GiB	0.37 GiB	25±22%
calme-2.3-rys-78b	IQ3_M	78.0B	35.33 GiB	0.38 GiB	36.83 GiB	0.37 GiB	25±22%
Hermes-4-70B	IQ4_XS	70.6B	35.33 GiB	0.35 GiB	36.81 GiB	0.39 GiB	25±22%
Llama-3.3-70B-Instruct	IQ4_XS	70.6B	35.33 GiB	0.35 GiB	36.81 GiB	0.39 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B	IQ4_XS	70.6B	35.33 GiB	0.35 GiB	36.81 GiB	0.39 GiB	25±22%
calme-2.4-llama3-70b	IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
calme-2.2-llama3-70b	IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
Rombos-LLM-70b-Llama-3.3	I1-IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
L3.3-Electra-R1-70b	I1-IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
L3.3-70B-Magnum-v4-SE	IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
Latxa-Llama-3.1-70B-Instruct-v2	I1-IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
Llama-3.1-Nemotron-70B-Instruct-HF	I1-IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
L3.3-70B-Euryale-v2.3	I1-IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
Hermes-3-Llama-3.1-70B	IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
Llama-3.1-70B	IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%
Anubis-70B-v1.2	IQ4_XS	70.6B	35.30 GiB	0.35 GiB	36.78 GiB	0.42 GiB	25±22%

●From the file◐Predictedwhat these mean

Speed is modeled, not measured: decode is memory-bandwidth bound, so tokens per second is bytes read per token against achievable bandwidth. Mixture-of-experts models carry a wider band because only the routed experts are read each step, and few have been measured publicly.

Questions people ask

What AI models can a A100 40GB run?: 2032 of 2118 indexed open-weight models fit a A100 40GB at 4,096 context with q4_0 KV cache, the largest being deepseek-llm-67b-chat at I1-Q4_0. That covers text, vision-language, image, video and speech models.
How much usable memory does a A100 40GB actually have?: Its nameplate is 40 GB, but about 37.20 GiB is available to a model once driver and compositor overhead is accounted for.
Is a A100 40GB fast for local AI?: Its memory bandwidth is 1555 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.