What AI models can a A100 40GB run?

2032 of 2118 indexed open-weight models fit a A100 40GB at 16,384 context with q4_0 KV cache, the largest being CodeLlama-70b-Python-hf at IQ4_XS. That covers text, vision-language, image, video and speech models.

How much usable memory does a A100 40GB actually have?

Its nameplate is 40 GB, but about 37.20 GiB is available to a model once driver and compositor overhead is accounted for.

Is a A100 40GB fast for local AI?

Its memory bandwidth is 1555 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.

NVIDIA · datacenter

A100 40GB

A100 40GB has 40 GB of VRAM at 1555 GB/s — about 37.20 GiB usable after driver and compositor overhead. 2032 of 2118 indexed models fit at 16K context with q4_0 KV.

Spec sheet· bandwidth, theoreticalFrom the file· fit from summed bytesPredicted· speed

Memory

40 GB

HBM2

Bandwidth

1555 GB/s

5120-bit bus

Tensor FP16

312 TF

dense

TDP

400 W

Context4K 8K 16K 32K 64K 128K

KV cachef16 q8_0 q4_0quantizing the KV cache is a ~2× lever on the dominant term at long context

text 1745vision language 183video 16audio tts 21image 2embedding 26audio asr 39

What fits at 16K context

largest quantization that fits, per model · 2032 of 2118 indexed

Model	Best quant	Params	Weights●	KV●	Total in memory◐	Headroom◐	tok/s◐
CodeLlama-70b-Python-hf	IQ4_XS	69.0B	34.64 GiB	1.41 GiB	37.17 GiB	0.03 GiB	25±22%
Midnight-Miqu-70B-v1.5	IQ4_XS	69.0B	34.64 GiB	1.41 GiB	37.17 GiB	0.03 GiB	25±22%
Qwen3.5-122B-A10BMoE	UD-IQ1_M	125B	36.02 GiB	0.11 GiB	37.15 GiB	0.05 GiB	141±37%
Meta-Llama-3-70B-Instruct	Q3_K_L	70.6B	34.60 GiB	1.41 GiB	37.13 GiB	0.07 GiB	25±22%
Maenad-70B	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B-Uncensored-v2-Unbiased-Reasoner	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
calme-2.4-llama3-70b	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
calme-2.2-llama3-70b	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Rombos-LLM-70b-Llama-3.3	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
L3.3-Electra-R1-70b	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
L3.3-70B-Magnum-v4-SE	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Latxa-Llama-3.1-70B-Instruct-v2	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Llama-3.3_70_b_uncensored_continued	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Llama-3.3-70B-Instruct-abliterated	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Strawberrylemonade-L3-70B-v1.2	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
grok-oss-Revenant-70B	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Llama-3.1-Nemotron-70B-Instruct-HF	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
L3.3-70B-Euryale-v2.3	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Hermes-4-70B-heretic	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Llama-3.3-70B-Instruct	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Llama-3.1-70B	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Hermes-3-Llama-3.1-70B	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Anubis-70B-v1.2	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Hermes-4-70B	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Golem-70B-v1b	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B-abliterated	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B-heretic	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
llama-3-firefunction-v2	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
DeepSeek-R1-Distill-Llama-70B	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Legion-V2.1-LLaMa-70B	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Assistant_Pepe_70B	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Tess-R1-Limerick-Llama-3.1-70B	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
SEMIKONG-70B	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
functionary-medium-v3.2KV unresolved	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Llama-3.1-WhiteRabbitNeo-2-70B	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Infinity-Instruct-7M-Gen-Llama3_1-70B	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
New-Dawn-Llama-3-70B-32K-v1.0	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Meta-Llama-3-70B-Instruct-abliterated-v3.5	I1-Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Athene-70B	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
L3.3-70B-Magnum-Diamond	Q3_K_L	70.6B	34.59 GiB	1.41 GiB	37.12 GiB	0.08 GiB	25±22%
Qwen3-Coder-NextMoE	Q3_K_M	79.7B	35.69 GiB	0.42 GiB	37.10 GiB	0.10 GiB	145±37%
GLM-4.5-Air-DerestrictedMoE	IQ1_M	110B	35.26 GiB	0.81 GiB	37.10 GiB	0.10 GiB	97±37%
GLM-4.5-AirMoE	IQ1_M	110B	35.26 GiB	0.81 GiB	37.09 GiB	0.11 GiB	97±37%
Qwen3-Next-80B-A3B-ThinkingMoE	Q3_K_M	81.3B	35.67 GiB	0.42 GiB	37.08 GiB	0.12 GiB	145±37%
Qwen3-Next-80B-A3B-InstructMoE	Q3_K_M	81.3B	35.67 GiB	0.42 GiB	37.08 GiB	0.12 GiB	145±37%
Hypernova-60B-2605MoE	I1-Q4_K_S	58.7B	35.89 GiB	0.15 GiB	37.03 GiB	0.17 GiB	119±37%
Qwen3.5-99BMoE	I1-IQ3_XXS	99.0B	35.87 GiB	0.11 GiB	37.01 GiB	0.19 GiB	132±37%
CalmeRys-78B-Orpo-v0.1	I1-IQ3_S	78.0B	34.33 GiB	1.51 GiB	36.97 GiB	0.23 GiB	25±22%
Qwen2.5-Coder-32B-Instruct	Q4_0	32.8B	34.72 GiB	1.13 GiB	36.94 GiB	0.26 GiB	25±22%
GLM-4.6VMoE	IQ2_XXS	108B	35.11 GiB	0.81 GiB	36.94 GiB	0.26 GiB	97±37%
gemma-4-31B-it-abliterated	Q4_K_M	31.3B	34.81 GiB	1.03 GiB	36.92 GiB	0.28 GiB	25±22%
calme-2.3-rys-78b	Q3_K_S	78.0B	34.24 GiB	1.51 GiB	36.88 GiB	0.32 GiB	25±22%
Llama-3_3-Nemotron-Super-49B-v1_5	Q3_K_L	49.9B	24.47 GiB	11.25 GiB	36.86 GiB	0.34 GiB	25±22%
Valkyrie-49B-v2.1	I1-Q3_K_L	49.9B	24.47 GiB	11.25 GiB	36.86 GiB	0.34 GiB	25±22%
Llama-3_3-Nemotron-Super-49B-v1	Q3_K_L	49.9B	24.47 GiB	11.25 GiB	36.86 GiB	0.34 GiB	25±22%
CodeLlama-70b-Instruct-hf	I1-IQ4_XS	69.0B	34.30 GiB	1.41 GiB	36.83 GiB	0.37 GiB	25±22%
Nous-Hermes-Llama2-70b	I1-IQ4_XS	69.0B	34.30 GiB	1.41 GiB	36.83 GiB	0.37 GiB	25±22%
Qwen3-53B-A3B-2507-THINKING-TOTAL-RECALL-v2-MASTER-CODERMoE	I1-Q5_K_M	53.0B	35.08 GiB	0.74 GiB	36.81 GiB	0.39 GiB	89±37%
Qwen3.5-88BMoE	I1-IQ3_S	87.7B	35.64 GiB	0.11 GiB	36.77 GiB	0.43 GiB	127±37%
deepseek-llm-67b-chat	IQ4_XS	67.4B	34.00 GiB	1.67 GiB	36.77 GiB	0.43 GiB	25±22%

●From the file◐Predictedwhat these mean

Speed is modeled, not measured: decode is memory-bandwidth bound, so tokens per second is bytes read per token against achievable bandwidth. Mixture-of-experts models carry a wider band because only the routed experts are read each step, and few have been measured publicly.

Questions people ask

What AI models can a A100 40GB run?: 2032 of 2118 indexed open-weight models fit a A100 40GB at 16,384 context with q4_0 KV cache, the largest being CodeLlama-70b-Python-hf at IQ4_XS. That covers text, vision-language, image, video and speech models.
How much usable memory does a A100 40GB actually have?: Its nameplate is 40 GB, but about 37.20 GiB is available to a model once driver and compositor overhead is accounted for.
Is a A100 40GB fast for local AI?: Its memory bandwidth is 1555 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.