LLM inference speed calculator
Single-stream tokens per second ≈ memory bandwidth × 0.6 ÷ model size. Pick a model and quantization to see how fast each GPU can decode it.
—Bandwidth × 0.6 ÷ weights — single stream, short prompt
Every GPU, this model
Updates as you change the calculator above
| GPU | Bandwidth | Tokens / second |
|---|---|---|
| RTX 3060 | 360 GB/s | — |
| RTX 4060 Ti | 288 GB/s | — |
| RTX 3090 | 936 GB/s | — |
| RTX 4090 | 1008 GB/s | — |
| RTX 5090 | 1792 GB/s | — |
| A100 | 1555 GB/s | — |
| A100 | 2039 GB/s | — |
| H100 SXM | 3350 GB/s | — |
| H200 | 4800 GB/s | — |
| B200 | 8000 GB/s | — |
| AMD MI300X | 5300 GB/s | — |
| Mac Studio M3 Ultra unified memory, lower bandwidth | 819 GB/s | — |
Popular models at GGUF Q4_K_M
Single-stream tok/s, short prompt
| Model | RTX 4090 | H100 SXM |
|---|---|---|
| Llama 3.1 8B | 125 tok/s | 416 tok/s |
| Llama 3.1 70B | 14 tok/s | 48 tok/s |
| Llama 3.1 405B | 2.5 tok/s | 8.2 tok/s |
| Qwen 2.5 7B | 143 tok/s | 476 tok/s |
| Qwen 2.5 72B | 14 tok/s | 46 tok/s |
| Mistral 7B | 143 tok/s | 476 tok/s |
Speed is a division, not a mystery.
Every generated token streams the entire model from VRAM through the compute units. That makes decode speed a simple conversion: memory bandwidth divided by model size, discounted by the share of peak bandwidth a real runtime achieves. Halve the model with a tighter quant and you double your tokens per second on the same card.
This is also why an RTX 4090 beats cards with more raw TFLOPS: at 1,008 GB/s it moves weights faster than almost anything outside a datacenter, and that bandwidth — not its FLOPS — sets your reading speed. Compute only becomes the limit during prompt processing and batched serving.
Frequently asked questions
Why is decoding memory-bound?
Generating each token requires reading every model weight from VRAM once. No matter how fast the compute cores are, they must wait for the weights to arrive — so tokens per second tracks memory bandwidth divided by model size. Batch inference improves utilization but per-stream speed still follows this law.
Why multiply bandwidth by 0.6?
Realistic local runtimes (llama.cpp, vLLM, ExLlama) achieve roughly 50-70% of peak memory bandwidth once attention, sampling, and OS overhead are counted. We use 60% — a community-typical middle — and state it openly instead of hiding it, so you can adjust the mental math either way.
How accurate is this estimate?
For dense models with short prompts it lands within roughly ±25% of measured llama.cpp numbers on RTX 3090/4090 class hardware. Long prompts pay a prefill cost and MoE models activate only part of their weights, so both run slower or faster than this simple law predicts.