Best Local LLM Hardware: questions & answers

It depends on model size. For models up to about 32B shared by a few people, our editorial choice is a 24 GB CUDA machine, Olares One, listed at 4,999 USD. For 70B and larger, a 128 GB unified-memory machine like a DGX Spark, or a Mac Studio, holds more.

At 4-bit, roughly 19-20 GB plus a few GB for context, by our approximate rule. A 24 GB GPU fits it with little room to spare.

Not on a single consumer GPU in our data. A dense 70B model at 4-bit needs roughly 40 GB by our approximate rule, and NVIDIA lists 32 GB for the RTX 5090, the largest consumer card here.

A Mac Studio M5 Ultra holds very large models, up to 512 GB of unified memory in Apple's largest configuration. An NVIDIA GPU runs models that fit in its memory faster and supports CUDA tools like vLLM. Pick by the size of model you need.

It is the cheapest 24 GB CUDA option in our data. One build guide estimates about 3,800 USD for a full tower with a used card. You build and maintain it yourself.

It suits large models. NVIDIA lists 128 GB of unified memory at 273 GB/s with CUDA, and it costs 6,950 USD. Models that fit in 24 GB of VRAM usually generate faster on a dedicated GPU.

For one person reading, anything faster than reading speed feels instant. For agents and shared servers, more is better. Judge shared machines by per-request speed under load.

Yes, by spilling part of the model from its 24 GB of VRAM into its 96 GB of system memory. Olares reports about 36 tokens per second for it in its own test.