Can I run a 70B model on an RTX 4090?

At 4-bit quantization a 70B model's weights alone are roughly 35 GB, which exceeds the RTX 4090's 24 GB, so a single 4090 cannot hold it fully in VRAM. Options are heavier quantization with quality loss, CPU/GPU split offloading (slower), or a smaller model.

Does Apple unified memory work for LLMs?

Yes — on Apple Silicon the GPU shares unified memory, and by default macOS lets the GPU use roughly 65-75% of total RAM. A 64 GB Mac can typically dedicate around 48 GB to model weights and KV cache, enough for a 70B model at 4-bit quantization.

LLM VRAM Calculator — can my GPU run it?

Q: How much VRAM does an LLM need?

A rough estimate: weight memory = parameter count × bytes per parameter (2 bytes at FP16, 1 at INT8/Q8, ~0.5 at 4-bit), plus KV cache that grows with context length, plus around 10% runtime overhead. A 7B model at Q4 needs roughly 4-5 GB for weights; a 70B model at Q4 roughly 35 GB plus KV cache.

Pick a model size, a quantization format and a context length. Get an estimated VRAM requirement and a yes/no against common GPUs and Apple unified memory.

Built by AI Model Hub · This is an estimator — the formula and every assumption are printed below the result

The estimate: VRAM ≈ weights (parameters × bytes per parameter) + KV cache (grows with context length) + ~10% runtime overhead. Real usage varies by runtime (llama.cpp, vLLM, MLX), attention implementation and driver — treat results as a planning estimate, not a guarantee.

1 · Model & settings

Model preset (optional)

Sizes from AI Model Hub model pages. For MoE models, all experts must sit in memory — use total parameters.

Parameters (billions)

e.g. 7 for a 7B model, 70 for a 70B

Quantization

GGUF "K-quant" sizes are approximations of their average bits/weight

Context length (tokens)

KV cache grows linearly with context you actually use

Advanced: architecture assumptions (editable)

Transformer layers

Auto-filled from a typical dense architecture for the chosen size

Hidden size

Auto-filled; edit if you know the real architecture

GQA ratio (heads ÷ KV heads)

Grouped-query attention shrinks KV cache by this factor

2 · Estimated VRAM

—

Weights

—

KV cache at selected context

—

Runtime overhead (~10%)

—

Estimated total

3 · Will it fit?

Hardware	Usable memory	Verdict	Headroom

Apple Silicon rows assume ~75% of unified memory is available to the GPU (the macOS default working-set limit); discrete GPUs assume the full listed VRAM minus nothing — your OS and display may already use 0.5-1 GB.

The formula (and its assumptions)

weights_GB = params_B × bytes_per_param
kv_GB      = 2 × layers × (hidden ÷ GQA_ratio) × 2 bytes × context_tokens ÷ 1024³
overhead   = 10% × weights_GB
total      = weights_GB + kv_GB + overhead

Weights. Parameter count × bytes per parameter. Quantization formats are averages: GGUF K-quants mix precisions, so "Q4 ≈ 0.5 bytes/param" is an approximation.
KV cache. 2 (key + value) × layers × per-token KV width × 2 bytes (FP16 cache) × context. Layer count and hidden size are auto-filled from typical dense-transformer shapes for the chosen parameter count (e.g. 7B → 32 layers × 4096 hidden; 70B → 80 × 8192) — real architectures differ, so edit them in Advanced if you know the model card. The GQA ratio divides KV width; many modern models use grouped-query attention.
Overhead. A flat ~10% of weights for runtime buffers, activations and fragmentation. Some runtimes need more.
MoE models. Mixture-of-experts models compute with few active parameters but still need all experts resident — size by total parameters, not active. (That is why DeepSeek V4-Flash's page lists a ~160 GB download for its 284B weights.)

Which tracked models can you actually get weights for?

Only open-weight models can be self-hosted. From the AI Model Hub index: DeepSeek V4 (MIT), Kimi K2.6 (Modified MIT) and Gemma 4 (Apache 2.0). Everything else — Claude, GPT, Gemini, Qwen 3.7-Max — is API-only; if that's your route, the API pricing calculator is the tool you want.

Frequently asked questions

How much VRAM does an LLM need?

Estimate weights as parameters × bytes per parameter (2 at FP16, 1 at INT8, ~0.5 at 4-bit), then add KV cache for your context length and ~10% overhead. A 7B at Q4 is roughly 3.5 GB of weights; a 70B at Q4 roughly 35 GB.

Can I run a 70B model on a 24 GB card (RTX 4090 / 3090)?

Not fully in VRAM: ~35 GB of Q4 weights exceeds 24 GB. You can offload layers to system RAM at a large speed cost, drop to more aggressive quantization with quality loss, or choose a ~30B-class model instead.

Is unified memory on a Mac as good as GPU VRAM?

It works well for local LLMs (the GPU addresses the same memory), with lower memory bandwidth than high-end discrete GPUs, so token throughput is typically lower. By default macOS caps the GPU working set at roughly 65-75% of total RAM — this calculator assumes 75%.

Why is my real usage different from the estimate?

Runtimes differ: attention implementation, KV-cache quantization (many runtimes can hold KV in 8-bit), batch size, parallel sequences and fragmentation all move the number. The formula here is a planning estimate with all assumptions stated above.

Model sizes referenced from AI Model Hub pages: Gemma 4, DeepSeek V4, Kimi K2.6, Mistral Small 4. GPU memory capacities are manufacturer specifications. All VRAM figures produced by this page are estimates from the stated formula.