Does functionary-medium-v3.2 fit in 8GB of VRAM?
Not at these settings. No indexed quantization of functionary-medium-v3.2 fits 8GB card at any context we compute, with q8_0 KV. The smallest shipped quantization is 15.60 GiB in weights alone, against 7.44 GiB usable. CPU offload can still run it, slowly.
Every quantization at every context
| Quant | Weights● | 4K◐ | 8K◐ | 16K◐ | 32K◐ | 64K◐ | 128K◐ |
|---|---|---|---|---|---|---|---|
| Q8_0 | 69.83 GiB | 71.4 | 72.1 | 73.4 | 76.1 | 81.4 | 92.0 |
| Q6_K | 53.91 GiB | 55.5 | 56.2 | 57.5 | 60.2 | 65.5 | 76.1 |
| Q5_K_M | 46.52 GiB | 48.1 | 48.8 | 50.1 | 52.8 | 58.1 | 68.7 |
| Q4_K_L | 40.33 GiB | 41.9 | 42.6 | 43.9 | 46.6 | 51.9 | 62.5 |
| Q4_K_M | 39.60 GiB | 41.2 | 41.9 | 43.2 | 45.8 | 51.2 | 61.8 |
| Q4_K_S | 37.58 GiB | 39.2 | 39.8 | 41.2 | 43.8 | 49.1 | 59.8 |
| IQ4_XS | 35.30 GiB | 36.9 | 37.6 | 38.9 | 41.5 | 46.9 | 57.5 |
| Q3_K_L | 34.59 GiB | 36.2 | 36.8 | 38.2 | 40.8 | 46.1 | 56.8 |
| Q3_K_M | 31.91 GiB | 33.5 | 34.2 | 35.5 | 38.2 | 43.5 | 54.1 |
| IQ3_M | 29.74 GiB | 31.3 | 32.0 | 33.3 | 36.0 | 41.3 | 51.9 |
| Q3_K_S | 28.79 GiB | 30.4 | 31.0 | 32.4 | 35.0 | 40.3 | 51.0 |
| IQ3_XXS | 25.58 GiB | 27.2 | 27.8 | 29.2 | 31.8 | 37.1 | 47.8 |
| Q2_K_L | 25.52 GiB | 27.1 | 27.8 | 29.1 | 31.8 | 37.1 | 47.7 |
| Q2_K | 24.56 GiB | 26.2 | 26.8 | 28.1 | 30.8 | 36.1 | 46.7 |
| IQ2_M | 22.46 GiB | 24.1 | 24.7 | 26.0 | 28.7 | 34.0 | 44.6 |
| IQ2_XS | 19.69 GiB | 21.3 | 21.9 | 23.3 | 25.9 | 31.2 | 41.9 |
| IQ2_XXS | 17.79 GiB | 19.4 | 20.0 | 21.4 | 24.0 | 29.3 | 40.0 |
| IQ1_M | 15.60 GiB | 17.2 | 17.9 | 19.2 | 21.8 | 27.2 | 37.8 |
Figures are GiB of total memory: weights plus KV cache plus compute buffer and backend overhead. Weights and KV are near-exact; the overhead term is modeled. Hover any cell for the breakdown.
Why there are no speeds on this page
A capacity is not a card. Whether a model fits depends only on memory, so every figure above holds for any 8GB accelerator. How fast it runs depends on memory bandwidth, which varies several-fold between cards of the same capacity — so putting a tokens-per-second number here would be inventing one. Pick a specific card from hardware and the speed column appears.
Why other calculators disagree
A parameters × bits ÷ 8 estimate ignores two things that dominate at long context. First, the weights themselves are not the nominal rate — quantizations are mixtures, so the real file is consistently larger than the label implies. Second, most of this model's layers cache only a 8,192-token window rather than the full context.