LLM VRAM calculator
Model size × quantization → VRAM. Enter parameters, pick a precision, and see inference or training memory plus which consumer and datacenter GPUs can run it.
—Weights + KV cache + runtime overhead
Which GPUs can run it?
Rows highlight as you change the calculator above
| GPU | VRAM | Runs this config |
|---|---|---|
| RTX 3060 | 12 GB | — |
| RTX 4060 Ti | 16 GB | — |
| RTX 3090 | 24 GB | — |
| RTX 4090 | 24 GB | — |
| RTX 5090 | 32 GB | — |
| A100 | 40 GB | — |
| A100 | 80 GB | — |
| H100 SXM | 80 GB | — |
| H200 | 141 GB | — |
| B200 | 180 GB | — |
| AMD MI300X | 192 GB | — |
| Mac Studio M3 Ultra unified memory, lower bandwidth | 512 GB | — |
Popular models × quantization
Inference GB at 4K context, overhead included
| Model | FP16 / BF16 | INT8 / Q8 | GGUF Q4_K_M |
|---|---|---|---|
| Llama 3.1 8B | 19.7 GB | 10.1 GB | 6.3 GB |
| Llama 3.1 70B | 169.3 GB | 85.3 GB | 52 GB |
| Llama 3.1 405B | 974.1 GB | 488.1 GB | 295.5 GB |
| Qwen 2.5 7B | 17 GB | 8.6 GB | 5.3 GB |
| Qwen 2.5 72B | 174.1 GB | 87.7 GB | 53.5 GB |
| Mistral 7B | 17.3 GB | 8.9 GB | 5.6 GB |
The formula is the conversion factor.
Weights need exactly parameters × bits-per-weight ÷ 8 bytes of memory — a 70B model at FP16 is 140 GB of weights, at INT4 it is 35 GB, before anything else runs. On top of that, inference adds roughly 20% for runtime buffers and activations, and each token of context holds a KV cache of 2 × layers × KV-heads × head-dim × 2 bytes.
Full fine-tuning multiplies the bill: mixed-precision AdamW keeps fp16 weights and gradients plus fp32 optimizer moments and master weights — about 16 bytes per parameter, 4× the FP16 inference cost. QLoRA exists precisely to collapse this back to near-inference cost by freezing 4-bit weights and training small adapters.
Frequently asked questions
How much VRAM does a 70B model need?
At FP16 the weights alone are 140 GB — about 169 GB with runtime overhead and a short context — which means A100/H100 80 GB class hardware with multi-GPU sharding. Quantized to INT4 it drops to roughly 43 GB total: two RTX 3090/4090 cards or a single A100 80 GB run it comfortably. GGUF Q4_K_M lands in between at about 52 GB.
Does the KV cache really matter?
At short contexts it is small next to the weights, but it grows linearly and silently. A 70B model holds about 0.33 GB per 1K tokens, so a 32K-token chat session adds over 10 GB on top of the model. Long-context work is where people unexpectedly run out of memory.
Why does fine-tuning need 4× the inference memory?
Mixed-precision AdamW keeps fp16 weights (2 bytes) and gradients (2 bytes) plus fp32 optimizer moments and master weights (12 bytes) — about 16 bytes per parameter versus 2 for FP16 inference. QLoRA collapses this by freezing 4-bit weights and training small adapters, bringing fine-tuning cost back near inference cost.