How much VRAM does Llama-2-70b-chat-hf need?

Q4_K_M is exactly 41,422,910,368 bytes (38.58 GiB) in weights. Add the KV cache, which depends on context length, plus roughly half a gigabyte of runtime overhead.

How large is Llama-2-70b-chat-hf's KV cache?

10.00 GiB at 32K context with an f16 cache, computed per layer from the model's architecture. Quantizing the cache to q8_0 roughly halves it.

meta-llama · text

Llama-2-70b-chat-hf

meta-llama/Llama-2-70b-chat-hf

Llama-2-70b-chat-hf at Q4_K_M is exactly 41,422,910,368 bytes (38.58 GiB / 41.42 GB) — an effective 4.804 bits per weight, not the nominal 4. Its KV cache at 32K is 10.00 GiB.

From the file· summed from 1 file(s)From the file· KV from mirror (mirror:NousResearch/Llama-2-70b-chat-hf)

Parameters

69.0B

Architecture

llama

80 layers

Context

4,096

native (config.json)

License

llama2

Shipped quantizations

● exact bytes, summed from published files

Quant	Size●	Exact bytes●	Effective bpw●	Tensors●	Publisher
Q2_K	27.27 GiB	29,279,253,408	3.396	—	TheBloke
Q3_K_S	27.86 GiB	29,919,294,368	3.470	—	TheBloke
Q5_K_S	28.47 GiB	30,574,407,680	3.546	—	TheBloke
Q3_K_M	30.91 GiB	33,186,657,184	3.849	—	TheBloke
Q3_K_L	33.67 GiB	36,147,835,808	4.192	—	TheBloke
Q4_0	36.20 GiB	38,872,249,248	4.508	—	TheBloke
Q4_K_S	36.39 GiB	39,073,575,840	4.532	—	TheBloke
Q4_K_M	38.58 GiB	41,422,910,368	4.804	—	TheBloke
Q5_0	44.20 GiB	47,461,397,408	5.505	—	TheBloke
Q5_K_M	45.41 GiB	48,753,767,328	5.654	—	TheBloke

KV cache by context

computed per layer

Context	KV cache (f16)●	Flat formula	Overstated by	Full / windowed / recurrent
4,096	1.25 GiB	1.25 GiB	—	80 / 0 / 0
8,192	2.50 GiB	2.50 GiB	—	80 / 0 / 0
16,384	5.00 GiB	5.00 GiB	—	80 / 0 / 0
32,768	10.00 GiB	10.00 GiB	—	80 / 0 / 0
65,536	20.00 GiB	20.00 GiB	—	80 / 0 / 0
131,072	40.00 GiB	40.00 GiB	—	80 / 0 / 0

Compare with

same modality, comparable size

vs Qwen3-Coder-30B-A3B-Instruct vs Qwen3.6-27B vs Qwen3.8-27B vs gemma-4-E4B-it vs Qwen3-30B-A3B-Thinking-2507 vs Qwen3-8B All alternatives →Will it fit on my card? →

Why other calculators give a different number

A parameters × bits ÷ 8 estimate puts Q4_K_M at roughly 36.13 GiB. The real file is 38.58 GiB, because a quantization is a mixture and some tensors are always kept at higher precision.

Architecture

from mirror:NousResearch/Llama-2-70b-chat-hf

Layers

Attention heads

KV heads

Head dim

128

Hidden size

8192

Vocab

32,000

Sliding window

none

SWA period

—

MLA

Experts

—

Experts per token

—

use_sliding_window

—

Questions people ask

How much VRAM does Llama-2-70b-chat-hf need?: Q4_K_M is exactly 41,422,910,368 bytes (38.58 GiB) in weights. Add the KV cache, which depends on your context length, plus roughly half a gigabyte of runtime overhead.
How large is Llama-2-70b-chat-hf's KV cache?: 10.00 GiB at 32K context with an f16 cache, computed per layer. Quantizing the cache to q8_0 roughly halves it, which is often the difference between a context length fitting and not.
Which quantization of Llama-2-70b-chat-hf should I use?: Q4_K_M is the usual default. Pick the largest quantization that fits your card at the context you actually need — the table above gives exact sizes for every one published.