How much VRAM do you need for a local LLM?
Last updated
How much VRAM you need for a local LLM depends on model size and precision. At 4-bit, plan roughly 0.5-0.6 GB per billion parameters plus a few GB for context: about 9 GB for 14B, 19-20 GB for 32B, 40 GB for 70B (approximate).
Olares One → Local LLM hardware → VRAM · quantization · tokens per second → The right machine for your model size
How much VRAM do you need for a local LLM?
How much VRAM you need for a local LLM depends on the model's size and the precision you run it at. At 4-bit, a model needs roughly 0.5-0.6 GB per billion parameters, plus a few GB for context. The rule of thumb is approximate, but it is close enough to size a purchase.
| Model size | Approx. memory at 4-bit | Fits in 24 GB of VRAM? |
|---|---|---|
| 8B | ~5 GB | Yes, with plenty left for long context |
| 14B | ~9 GB | Yes |
| 32B | ~19-20 GB | Yes, with little room left for context |
| 70B | ~40 GB | No |
| 120B | ~65 GB | No |
The weights have to sit in memory the GPU reads quickly. Once part of a model spills into system RAM, the slower link between CPU memory and GPU sets the pace.
What else takes up VRAM besides the model?
The KV cache takes up VRAM besides the model weights. For every token in a conversation, the model stores keys and values so it doesn't recompute them. That cache grows with context length and with the number of requests running at once, so long documents and extra users each claim more memory.
On a 24 GB card, a 32B model at 4-bit leaves only a few GB for the cache. You trade context length against parallel users. A 14B model leaves far more room. An 8-bit file stores each weight in twice the bits of a 4-bit file and needs roughly twice the memory, which quantization explained covers in detail.
Which hardware sits in each VRAM tier?
Local LLM hardware falls into three memory tiers in our data. Olares One and a used RTX 3090 tower have 24 GB of VRAM. NVIDIA's RTX 5090 desktop card has 32 GB. Unified-memory machines from Apple, NVIDIA and makers using AMD's Ryzen AI Max+ 395 range from 36 GB to 512 GB, depending on model and configuration.
At 24 GB, Olares One lists at 4,999 USD as a complete box, and one build guide estimates about 3,800 USD for a used RTX 3090 tower. The RTX 5090 card costs 1,999 USD on its own, and NVIDIA lists 1000 W of required system power for the PC around it. In the unified tier, the DGX Spark and ASUS Ascent GX10 have 128 GB, the Mac Studio M5 Ultra starts at 96 GB, and the Ryzen boxes have 128 GB of which the GPU can use only part. Unified memory or VRAM weighs the trade-off.
Is 24 GB of VRAM enough for most people?
Yes. In our view 24 GB of VRAM is enough for most people running local models, because it holds any model up to about 32B at 4-bit. We'd change that view for anyone whose main model is a dense 70B, which needs roughly 40 GB at 4-bit by the approximate rule.
On Olares One, Qwen3-30B-A3B ran at 157 tokens per second with a single request on vLLM, in Olares' own test. Larger models can use its 96 GB of system RAM as overflow, and Olares reports about 36 tokens per second for gpt-oss-120b run that way. For 70B, read what hardware a 70B model needs.
Questions & answers
Is 24 GB of VRAM enough for a 32B model?
Yes, at 4-bit. A 32B model needs roughly 19-20 GB by our approximate rule, which leaves a few GB for context. For long contexts or many users at once, a 14B model or a mixture-of-experts model leaves more headroom.
Does system RAM count toward VRAM?
No. Some engines can place part of a model in system RAM, but those layers run much slower. On unified-memory machines such as a Mac Studio or DGX Spark, the GPU uses the shared memory directly.
How much VRAM does context take?
It depends on the model's architecture, the context length and the number of parallel requests. Our rule of thumb allows a few GB for context on top of the weights.
Sources for this entry
- Olares One FAQ (Olares docs) (Company-published)
- Olares blog: local AI hardware benchmarking (Oct 2025) (Company-published)
- Olares One specifications (Olares docs) (Company-published)
- NVIDIA GeForce RTX 5090 (desktop card): maker's page (Maker-published)
- Used RTX 3090 tower build: maker's page (Maker-published)
- Apple Mac Studio (M5 Ultra): maker's page (Maker-published)
- NVIDIA DGX Spark: maker's page (Maker-published)
Continue this reading path
How the related websites connect
Cite this entry
Best Local LLM Hardware. “How much VRAM do you need for a local LLM?.” Content reviewed 13 November 2026.
https://bestlocalllmhardware.com/how-much-vram-for-local-llm/