How much VRAM do you need to run an LLM locally?

You want to run an LLM on your own machine. The first question is always the same: will it fit in memory?

The answer comes down to simple math. Parameter count, quantization, and context length. Once you know how these three combine, you can look at any model and know if your GPU can run it.

Let’s work through it.

The weights

An LLM is mostly a big pile of numbers, the weights. Each parameter is one number, and how much space it takes depends on the precision you store it at.

At full precision (FP16), each parameter takes 16 bits, which is 2 bytes. So the formula is:

weights in GB = parameters (in billions) × bits ÷ 8

An 8B model at FP16:

8 × 16 ÷ 8 = 16 GB

16 GB just for the weights. That doesn’t fit on most consumer GPUs. This is why almost nobody runs local models at full precision.

Quantization

Quantization stores each weight with fewer bits. Instead of 16, you use 8, or 4. You lose a little quality, but the model shrinks a lot.

Same 8B model at different precisions:

FP16: 8 × 16 ÷ 8 = 16 GB
Q8:   8 × 8 ÷ 8  = 8 GB
Q4:   8 × 4 ÷ 8  = 4 GB

Q8 is nearly indistinguishable from FP16 in practice. Q4 (the Q4_K_M variant you see everywhere) is the usual sweet spot for local runs: 4× smaller than FP16, with a quality drop most people don’t notice. Below that, Q3 starts losing noticeably on hard tasks, and Q2 is often incoherent.

My advice: start with Q4 and only go up if you see quality problems.

The KV cache

Weights are not the whole story. While the model generates text, it keeps a KV cache in memory: intermediate values for every token in the context, so it doesn’t recompute them on each new token.

The KV cache grows with context length. Longer conversations, bigger documents, more memory.

The exact size depends on the model architecture, but a useful rule of thumb is about 75 MB per 1,000 context tokens per billion parameters (at FP16-equivalent precision, scaled down by quantization).

For an 8B model at Q4 (so scaled by 4/16), that’s:

8 × 75 × (4/16) = 150 MB per 1k tokens

At 8,000 tokens of context:

8 × 150 MB = 1,200 MB ≈ 1.2 GB

Not huge for a small model. But watch what happens with a 70B model at Q4:

70 × 75 × (4/16) = 1,312 MB per 1k tokens

At 8k context that’s over 10 GB, just for the cache. Context length is the hidden VRAM cost people forget about.

Overhead

The runtime (llama.cpp, Ollama, MLX) adds its own overhead on top: buffers, compute scratch space, framework bookkeeping. A 1.1× multiplier on the weights is a reasonable estimate.

Putting it all together for an 8B model at Q4 with 8k context:

weights:  8 × 4 ÷ 8 × 1.1 = 4.4 GB
KV cache: 8k × 150 MB/1k  = 1.2 GB
total:                      5.6 GB

That fits comfortably on a 12 GB card.

I built a small calculator that does this math for any model size, quantization, and context length, and checks the result against common GPUs and Macs: the LLM VRAM calculator.

What fits where

One more thing before matching models to hardware: you never get 100% of the memory.

On a discrete GPU, the driver and display eat some VRAM. Count on about 90% usable. On Apple Silicon, the memory is unified: the GPU shares it with macOS and your apps, so about 70–75% is realistically available for the model.

With that in mind:

• RTX 3060 (12 GB) — about 10.8 GB usable. Runs 7–8B at Q4 with room to spare.
• RTX 4060 Ti (16 GB) — about 14.4 GB usable. Comfortable for 13–14B at Q4.
• RTX 4090 (24 GB) — the enthusiast default. 32B at Q4 fits, 70B does not.
• RTX 5090 (32 GB) — more headroom, but a 70B Q4 still needs ~38.5 GB of weights alone. Not enough.
• Mac with 16 GB unified — about 11.5 GB usable. Same class as the 3060: 7–8B at Q4.
• Mac with 64 GB unified — about 46 GB usable. A 70B at Q4 fits at short context, but gets tight fast as the KV cache grows.
• Mac with 96 GB or more — 70B at Q4 with real context headroom.

Let’s verify that 70B case with the formula. Weights at Q4:

70 × 4 ÷ 8 × 1.1 = 38.5 GB

Add 8k of context (10.3 GB of KV cache) and you’re at ~48.8 GB. That’s past the 46 GB usable on a 64 GB Mac. Drop to 4k context and you squeeze in at ~43.6 GB, tight. This is why “it fits” always depends on how much context you actually use.

The short version

Three numbers decide everything:

1. Parameters × bits ÷ 8 gives you the weights.
2. Context length grows the KV cache, and it’s not negligible on big models.
3. Multiply weights by ~1.1 for runtime overhead, and remember you only get 90% of a GPU (or ~72% of a Mac’s unified memory).

And if a model doesn’t fit and you fall back to an API, keep the provider swappable in your code. I wrote about how in an LLM adapter pattern for Cloudflare Workers. This calculator is part of a bigger set, too: I recently shipped 90 free browser tools for developers, all running client-side.

Related posts about ai: