Tokens per second: what speed you need
Last updated
Tokens per second measures how fast a model writes. For one person chatting, a speed faster than you read feels instant. Agents, coding tools and shared servers need more. Check single-request speed, per-request speed under load and time to first token.
Olares One → Local LLM hardware → VRAM · quantization · tokens per second → The right machine for your model size
Tokens per second: what speed do you need?
The tokens per second you need depends on who reads the output. One person chatting needs text to appear faster than they read, and past that point extra speed is hard to notice. Coding agents and shared servers need much more, because agents chain many calls and a server splits its capacity across everyone asking at once.
A token is a piece of a word. Engines count speed in tokens because that is the unit the model writes.
Which speed numbers should you check?
Check time to first token, single-request generation speed and per-request speed under load. Time to first token reflects prefill, the step that reads your whole prompt, and grows with long documents and code files. Single-request speed is the figure most reviews quote. Per-request speed with several streams running is what users of a shared machine feel.
Multiply per-request speed by the number of streams and you get total throughput. That tells you capacity, but each person only sees their own stream.
What sets a machine's tokens per second?
Memory bandwidth sets generation speed, because each new token reads the model's active weights from memory once. Faster memory gives more tokens per second. Smaller or more heavily quantized models run faster. Mixture-of-experts models run fast for their size, since each token reads only a few experts.
Prefill is different. It processes the whole prompt in parallel and leans on GPU compute. The serving engine matters as soon as a second request arrives, and vLLM's continuous batching reads the weights once per step to advance every active request (quantization explained covers the bytes side).
How fast is Olares One in Olares' own test?
In Olares' own test, Olares One ran Qwen3-30B-A3B at 157 tokens per second with a single request on vLLM and about 81 tokens per second per request with 8 concurrent requests. With Ollama the same model ran at 107 tokens per second alone and 32 per request at 8 concurrent.
The gap between engines under load is the lesson. Our view is that a shared machine should be judged by per-request speed at your expected concurrency, and single-request speed matters only when one person uses it. Olares also reports about 36 tokens per second for gpt-oss-120b spilling into system RAM, fine for one reader and slow for a team. See hardware for an inference server.
Questions & answers
Why is my model fast at first and slower in long chats?
Each new token attends to the whole context, and the KV cache grows with it. Long contexts also lengthen prefill, so the first token arrives later.
Is total throughput or per-user speed more important?
Per-user speed, for anything people read. Total throughput matters when you plan how many users one machine can carry.
Sources for this entry
- Olares blog: local AI hardware benchmarking (Oct 2025) (Company-published)
- Olares One FAQ (Olares docs) (Company-published)
Continue this reading path
How the related websites connect
Cite this entry
Best Local LLM Hardware. “Tokens per second: what speed you need.” Content reviewed 13 November 2026.
https://bestlocalllmhardware.com/tokens-per-second-explained/