Hardware for a local LLM inference server
Last updated
Hardware for a local LLM inference server needs VRAM for the model plus a KV cache per parallel request, a batching engine such as vLLM, and a box built to stay on. Up to about 32B, one 24 GB CUDA GPU serves a small team.
Olares One → Local LLM hardware → VRAM · quantization · tokens per second → The right machine for your model size
What hardware does a local LLM inference server need?
Hardware for a local LLM inference server needs GPU memory for the weights plus a KV cache for every active request, enough GPU compute for batching, and a chassis that runs all day quietly. System RAM, storage and a wired network matter less, though any one of them can still become the bottleneck.
System RAM holds models parked between uses and absorbs overflow. Storage holds model files of many GB each, and switching models reads them from disk. A wired link keeps streamed tokens steady.
How does batching change what the hardware can serve?
Batching lets one GPU serve many requests with little extra memory traffic. A single request leaves most of the GPU's compute idle, because generation waits on memory reads. A batching engine like vLLM reads the weights once per step and advances every active request with that read, so total throughput rises while each request slows a little.
In Olares' own test on Olares One with Qwen3-30B-A3B, vLLM ran at 157 tokens per second alone and about 81 per request at 8 concurrent. Ollama ran at 107 alone and 32 per request at 8. We treat the engine as part of the hardware decision, because on the same GPU vLLM served eight users far better. Ollama still fits a single user. The terms are defined in tokens per second explained.
How much memory does each user need?
Each concurrent user needs room for their own KV cache, which grows with their context length. With a 32B model at 4-bit on a 24 GB GPU, roughly 19-20 GB goes to weights by our approximate rule. Long contexts and many users then compete for the few GB left.
A smaller model, or a mixture-of-experts model that finishes requests sooner, often serves a team better than the largest model that loads. If one model must serve many long-context users, a 32 GB card or a 128 GB machine gives more headroom. See how much VRAM you need.
What happens when one box isn't enough?
When one Olares One isn't enough, two or more Olares devices can form a cluster, according to Olares' FAQ. The cluster raises throughput and allows larger models through vLLM pipeline parallelism. It doesn't make a single request faster, and it only joins devices of the same architecture running the same OS.
Its Ethernet port runs at 2.5 Gbps, a limit Olares attributes to the hardware platform vendor. The Thunderbolt 5 port also takes an external GPU enclosure. For models beyond one box, compare hardware for a 70B model.
What does an always-on server cost in power and noise?
Olares One draws 30 W in standby and uses a 330 W power adapter, according to Olares. Olares also states 19 dB at idle and under 39 dB at full load, measured in its own lab. Remote KVM isn't supported, so someone needs physical access to recover the box if the OS fails to boot.
Questions & answers
How many users can one 24 GB GPU serve?
It depends on model size, context length and engine. In Olares' own test, Olares One kept about 81 tokens per second per request with 8 concurrent requests on Qwen3-30B-A3B using vLLM.
Should an inference server use vLLM or Ollama?
vLLM, for several users at once. Olares' own test on the same box gave about 81 tokens per second per request at 8 concurrent with vLLM and 32 with Ollama, on Qwen3-30B-A3B. Ollama is simpler for one user and quick model switching.
Sources for this entry
- Olares blog: local AI hardware benchmarking (Oct 2025) (Company-published)
- Olares One FAQ (Olares docs) (Company-published)
- Olares One specifications (Olares docs) (Company-published)
Continue this reading path
How the related websites connect
Cite this entry
Best Local LLM Hardware. “Hardware for a local LLM inference server.” Content reviewed 13 November 2026.
https://bestlocalllmhardware.com/local-llm-inference-server-hardware/