What AI models can a A100 40GB run?

2032 of 2118 indexed open-weight models fit a A100 40GB at 4,096 context with q8_0 KV cache, the largest being Qwen3.5-88B at I1-IQ3_M. That covers text, vision-language, image, video and speech models.

How much usable memory does a A100 40GB actually have?

Its nameplate is 40 GB, but about 37.20 GiB is available to a model once driver and compositor overhead is accounted for.

Is a A100 40GB fast for local AI?

Its memory bandwidth is 1555 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.

NVIDIA · datacenter

A100 40GB

A100 40GB has 40 GB of VRAM at 1555 GB/s — about 37.20 GiB usable after driver and compositor overhead. 2032 of 2118 indexed models fit at 4K context with q8_0 KV.

Spec sheet· bandwidth, theoreticalFrom the file· fit from summed bytesPredicted· speed

Memory

40 GB

HBM2

Bandwidth

1555 GB/s

5120-bit bus

Tensor FP16

312 TF

dense

TDP

400 W

Context4K 8K 16K 32K 64K 128K

KV cachef16 q8_0 q4_0quantizing the KV cache is a ~2× lever on the dominant term at long context

text 1745vision language 183video 16audio tts 21image 2embedding 26audio asr 39

What fits at 4K context

largest quantization that fits, per model · 2032 of 2118 indexed

Model	Best quant	Params	Weights●	KV●	Total in memory◐	Headroom◐	tok/s◐
Qwen3.5-88BMoE	I1-IQ3_M	87.7B	36.10 GiB	0.05 GiB	37.18 GiB	0.02 GiB	127±37%
CalmeRys-78B-Orpo-v0.1	I1-IQ3_M	78.0B	35.33 GiB	0.71 GiB	37.17 GiB	0.03 GiB	25±22%
calme-2.3-rys-78b	IQ3_M	78.0B	35.33 GiB	0.71 GiB	37.17 GiB	0.03 GiB	25±22%
Hunyuan-A13B-InstructMoE	Q3_K_M	80.4B	35.91 GiB	0.27 GiB	37.17 GiB	0.03 GiB	24±22%
Hermes-4-70B	IQ4_XS	70.6B	35.33 GiB	0.66 GiB	37.12 GiB	0.08 GiB	25±22%
Llama-3.3-70B-Instruct	IQ4_XS	70.6B	35.33 GiB	0.66 GiB	37.12 GiB	0.08 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B	IQ4_XS	70.6B	35.33 GiB	0.66 GiB	37.12 GiB	0.08 GiB	25±22%
xLAM-8x7b-rMoE	Q6_K_L	46.7B	35.80 GiB	0.27 GiB	37.11 GiB	0.09 GiB	46±37%
Qwen3.5-122B-A10BMoE	UD-IQ1_M	125B	36.02 GiB	0.05 GiB	37.09 GiB	0.11 GiB	143±37%
Maenad-70B	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B-Uncensored-v2-Unbiased-Reasoner	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
calme-2.4-llama3-70b	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
calme-2.2-llama3-70b	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Rombos-LLM-70b-Llama-3.3	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
L3.3-Electra-R1-70b	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
L3.3-70B-Magnum-v4-SE	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Latxa-Llama-3.1-70B-Instruct-v2	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Llama-3.3_70_b_uncensored_continued	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Llama-3.3-70B-Instruct-abliterated	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
grok-oss-Revenant-70B	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Llama-3.1-Nemotron-70B-Instruct-HF	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
L3.3-70B-Euryale-v2.3	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Hermes-3-Llama-3.1-70B	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Hermes-4-70B-heretic	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Llama-3.1-70B	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Anubis-70B-v1.2	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Golem-70B-v1b	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B-abliterated	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B-heretic	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
llama-3-firefunction-v2	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Legion-V2.1-LLaMa-70B	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Assistant_Pepe_70B	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Tess-R1-Limerick-Llama-3.1-70B	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
SEMIKONG-70B	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
functionary-medium-v3.2KV unresolved	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Llama-3.1-WhiteRabbitNeo-2-70B	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Infinity-Instruct-7M-Gen-Llama3_1-70B	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
New-Dawn-Llama-3-70B-32K-v1.0	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Meta-Llama-3-70B-Instruct-abliterated-v3.5	I1-IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Athene-70B	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
L3.3-70B-Magnum-Diamond	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
Meta-Llama-3-70B-Instruct	IQ4_XS	70.6B	35.30 GiB	0.66 GiB	37.09 GiB	0.11 GiB	25±22%
dolphin-2.6-mixtral-8x7bMoE	I1-Q6_K	46.7B	35.74 GiB	0.27 GiB	37.05 GiB	0.15 GiB	46±37%
Nous-Hermes-2-Mixtral-8x7B-DPOMoE	Q6_K	46.7B	35.74 GiB	0.27 GiB	37.05 GiB	0.15 GiB	46±37%
Mixtral-8x7B-Instruct-v0.1MoE	Q6_K	46.7B	35.74 GiB	0.27 GiB	37.05 GiB	0.15 GiB	46±37%
Open_Gpt4_8x7B_v0.1MoE	Q6_K	46.7B	35.74 GiB	0.27 GiB	37.05 GiB	0.15 GiB	46±37%
dolphin-2.5-mixtral-8x7bMoE	Q6_K	46.7B	35.74 GiB	0.27 GiB	37.05 GiB	0.15 GiB	46±37%
dolphin-2.7-mixtral-8x7bMoE	Q6_K	46.7B	35.74 GiB	0.27 GiB	37.05 GiB	0.15 GiB	46±37%
Mixtral-8x7B-v0.1MoE	Q6_K	46.7B	35.74 GiB	0.27 GiB	37.05 GiB	0.15 GiB	46±37%
Mixtral-8x7B-MoE-RP-StoryMoE	Q6_K	46.7B	35.74 GiB	0.27 GiB	37.05 GiB	0.15 GiB	46±37%
Open_Gpt4_8x7B_v0.2MoE	Q6_K	46.7B	35.74 GiB	0.27 GiB	37.05 GiB	0.15 GiB	46±37%
GLM-4.5-AirMoE	UD-TQ1_0	110B	35.63 GiB	0.38 GiB	37.04 GiB	0.16 GiB	104±37%
Huihui-GLM-4.7-Flash-abliterated-57BMoE	I1-Q5_K_S	57.3B	35.72 GiB	0.28 GiB	37.04 GiB	0.16 GiB	106±37%
GLM-4.5VMoE	I1-IQ2_XXS	108B	35.61 GiB	0.38 GiB	37.02 GiB	0.18 GiB	104±37%
Qwen3-Coder-NextMoE	UD-IQ4_XS	79.7B	35.79 GiB	0.20 GiB	36.98 GiB	0.22 GiB	155±37%
Hypernova-60B-2605MoE	I1-Q4_K_S	58.7B	35.89 GiB	0.08 GiB	36.96 GiB	0.24 GiB	121±37%
Qwen3.5-99BMoE	I1-IQ3_XXS	99.0B	35.87 GiB	0.05 GiB	36.95 GiB	0.25 GiB	134±37%
Rombo-LLM-V3.0-Qwen-72b	I1-Q3_K_M	72.7B	35.11 GiB	0.66 GiB	36.90 GiB	0.30 GiB	25±22%
Qwen2.5-72B-Instruct-abliterated	I1-Q3_K_M	72.7B	35.11 GiB	0.66 GiB	36.90 GiB	0.30 GiB	25±22%
Qwen2.5-72B-Instruct-abliterated-v2	I1-Q3_K_M	72.7B	35.11 GiB	0.66 GiB	36.90 GiB	0.30 GiB	25±22%

●From the file◐Predictedwhat these mean

Speed is modeled, not measured: decode is memory-bandwidth bound, so tokens per second is bytes read per token against achievable bandwidth. Mixture-of-experts models carry a wider band because only the routed experts are read each step, and few have been measured publicly.

Questions people ask

What AI models can a A100 40GB run?: 2032 of 2118 indexed open-weight models fit a A100 40GB at 4,096 context with q8_0 KV cache, the largest being Qwen3.5-88B at I1-IQ3_M. That covers text, vision-language, image, video and speech models.
How much usable memory does a A100 40GB actually have?: Its nameplate is 40 GB, but about 37.20 GiB is available to a model once driver and compositor overhead is accounted for.
Is a A100 40GB fast for local AI?: Its memory bandwidth is 1555 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.