Experiment · Local inference
One GPU, three ways to serve a model
Earlier today we streamed a 188 GB model from an SSD (that post). Then we asked the obvious next question: on the same 72 GB card, what do the other ways of serving a model look like, measured the same way? Three setups, one benchmark, one afternoon. The answer is that “fastest” has three different winners: one for a single person typing, one for reading long prompts, and one for a classroom full of people asking at once.
The result
The three setups
Same machine for all three: NVIDIA RTX PRO 5000 72GB Blackwell, AMD Ryzen 7 9800X3D, 96 GB of RAM, a Samsung 9100 PRO NVMe drive. Each engine had the card to itself while it ran.
- vLLM 0.31, Qwen3.8-27B in FP8. The dense 27-billion-parameter model, FP8 weights and FP8 key-value cache, 196K context, up to 12 requests batched together, MTP speculative decoding (4 draft tokens), prefix caching.
- llama.cpp, Qwen3.8-Flash-Next at 4.27 bits per weight. AtomicChat's mixed quantisation of the big mixture-of-experts model, small enough to sit entirely on the card. Two slots of 196K context, q8_0 key-value cache, MTP drafts.
- Strata 0.1.58, Qwen3.8-Flash-Next at Q8_0. The same mixture-of-experts model at near-full precision: 188 GB, so 66 GiB of experts stay in RAM and the rest is read from the SSD while it answers. Two slots, 262K context, int8 key-value cache, MTP drafts.
Read that list carefully, because it is the main caveat of this post: these are two different models at three different precisions. The 27B is a smaller dense model; Flash-Next is a much larger mixture-of-experts model. This compares ways of serving on one card, not engines in a vacuum. Nobody gets to say “vLLM is seven times faster than Strata” on the strength of this page.
One benchmark for every engine
Each engine reports its own speed in its own way, and vLLM reports none per request. So we wrote one small benchmark script, llm-bench, that talks to any OpenAI-compatible server and measures everything from the client side, from the stream of tokens, identically for all three. The workload is the one from Strata's community quick suite: three short chat questions, fresh prompts of 4K, 16K and 32K tokens, a six-turn coding-agent session with four tools that keeps resending the growing conversation, bursts of 1, 4 and 12 simultaneous requests, and a needle-in-a-haystack recall test. Thinking off, temperature 0, three runs per test, medians.
Does a client stopwatch agree with the engine's own? We checked: within 0.1 percent of llama.cpp's own timings, within 0.5 percent of Strata's, and the whole Strata run landed within 1 to 3 percent of Strata's own quick suite run that morning. Good enough to put three engines side by side.
The script, the raw results of every run on this page and the exact settings are on GitHub: Jan-byterider/llm-bench (MIT). One Python file, no dependencies; point it at your own server and compare.
One person at a time
| Test (one request at a time) | vLLM 27B FP8 | llama.cpp Flash-Next 4.27 bpw | Strata Flash-Next Q8 |
|---|---|---|---|
| chat decode, tok/s | 72.9 | 133.1 | 56.7 |
| chat first token, s | 0.08 | 0.17 | 0.39 |
| 4K prompt, tok/s (first token) | 5,836 (0.69 s) | 2,403 (1.67 s) | 1,169 (3.44 s) |
| 16K prompt, tok/s (first token) | 5,516 (2.91 s) | 2,440 (6.58 s) | 2,348 (6.84 s) |
| 32K prompt, tok/s (first token) | 4,812 (6.66 s) | 2,248 (14.26 s) | 2,519 (12.72 s) |
| agent turn, first token, s | 0.89 | 1.99 | 3.22 |
| agent turn, decode tok/s | 97.8 | 142.1 | 108.4 |
Two different kinds of fast. Flash-Next on llama.cpp writes fastest: 133 tokens per second in chat and 142 in the agent turns, because a mixture-of-experts model only touches a small slice of its weights per token, and at 4.27 bits all of them are already on the card. The dense 27B on vLLM reads fastest: more than twice the prompt speed of either Flash-Next setup, so the first word of an answer arrives two to five times sooner. For a coding agent that rereads a growing conversation every turn, that is the number you feel.
Strata, streaming its experts from the SSD, is the slowest of the three for a single user, and that is the price of running the model at Q8_0 on a machine that cannot hold it. At 32K it overtakes llama.cpp on prompt reading, which fits what we saw this morning: bigger prompts mean more sequential SSD reads.
Twelve people at once
| Requests at once | vLLM 27B FP8 | llama.cpp Flash-Next 4.27 bpw | Strata Flash-Next Q8 |
|---|---|---|---|
| 1: total tok/s (slowest first token) | 60 (0.09 s) | 92 (0.58 s) | 52 (0.75 s) |
| 4: total tok/s (slowest first token) | 235 (0.16 s) | 134 (2.37 s) | 72 (4.58 s) |
| 12: total tok/s (slowest first token) | 566 (0.26 s) | 130 (10.28 s) | 76 (17.62 s) |
This is where the picture flips. vLLM batches every waiting request into the same pass over the weights, so adding users barely slows anyone down: with twelve at once, each still gets about 56 tokens per second and the slowest first word arrives in a quarter of a second. Both Flash-Next servers were configured with two slots, so from the third request on, people wait in line: with twelve at once the unluckiest user waits 10 seconds on llama.cpp and 18 on Strata before anything appears. (“Slowest” here is the 95th percentile.)
That is a configuration choice as much as an engine property; more slots would change those two rows, at the cost of memory for context. But it is exactly the question we care about when a class of students points their laptops at one machine.
Memory: all three found the needle
A code word hidden at 10, 50 and 90 percent of a 32K and a 128K prompt, asked back at the end. All three setups found it every time, 18 out of 18. At 128K tokens the full answer took 42 to 46 seconds on vLLM, 43 to 51 on Strata and about 84 on llama.cpp. Quantising Flash-Next to 4.27 bits did not cost it its long-context recall, at least not on this test.
Bonus: the same Q8 file on llama.cpp
If Strata can stream Q8_0 from the SSD, can plain llama.cpp do it too? It can
split a mixture-of-experts model the same three ways: the shared layers and
the experts of 23 layers on the GPU, the experts of the other 25 layers in
RAM (--n-cpu-moe 25), and the operating system reading from the
SSD whatever RAM cannot hold, because the file is memory-mapped. Same Unsloth
Q8_0 file as the Strata run, 32 seconds to load. A short run this time, one
pass per test, so read it as indicative.
| Same Q8_0 file, one request | Strata (streams experts from NVMe) | llama.cpp (experts split over GPU, RAM, SSD) |
|---|---|---|
| chat decode, tok/s | 56.7 | 21.9 |
| chat first token, s | 0.39 | 1.69 |
| 4K prompt, tok/s (first token) | 1,169 (3.44 s) | 508 (7.92 s) |
| agent turn, first token, s | 3.22 | 12.47 |
| agent turn, decode tok/s | 108.4 | 28.9 |
It works, and 22 tokens per second is usable for chat. But Strata is 2.5 to 4 times faster on the same file. In llama.cpp the experts that live in RAM are computed by the 8-core CPU, and anything that falls out of RAM comes back from the SSD one page fault at a time. Strata keeps the most used experts resident, fetches the rest from NVMe itself and pulls what each token needs to the GPU. Twelve seconds before every agent turn starts answering rules the llama.cpp split out for a coding agent; with more RAM, enough to hold all the experts, the comparison could look quite different.
So which one?
- One person, fast typing, big model: Flash-Next on llama.cpp. The most tokens per second and the larger model, as long as you are alone or almost alone.
- Agents and long prompts, or a room full of people: the 27B on vLLM. It reads prompts more than twice as fast and keeps twelve users responsive at once. This is what our classroom setup runs.
- The highest precision this card can serve: Strata at Q8_0. Slower, but it runs a model more than twice the size of the card, Q8_0 is the closest to full precision of the three, and on this machine it serves Q8_0 2.5 to 4 times faster than llama.cpp does.
The honest footnotes
- Different models. The 27B and Flash-Next are not the same model, and we did not measure answer quality here beyond the needle test.
- Not a clean lab. The machine is also our daily coding server; normal traffic and background services were not stopped during the runs.
- Small samples. Three runs per test, medians shown; ranges are in the raw results.
- Settings matter. vLLM ran with 4 MTP draft tokens and up to 12 sequences; both Flash-Next servers with 2 slots. Other settings give other numbers, especially in the concurrency table.