Experiment · Model comparison

Qwen3.8-27B at home against Mistral Large 4 in the cloud

24 October 2026 · By Jan Janssens · Read time: 6 min · ← All experiments

We run Qwen3.8-27B on our own hardware. Mistral Large 4 is the hosted model people compare against when they ask whether a local setup is good enough. So we gave both the same six short prompts and the same eight coding problems with hidden unit tests. On code they finished level. With thinking off the two were within a second of each other on time to a finished answer, and with thinking on the local model was about twice as quick.

This is a small test, not a leaderboard entry. Six prompts and sixteen coding samples per model are enough to see a pattern and not enough to rank anything. The caveats are at the end and they matter.

The result

How we set it up

Qwen3.8-27B is an open-weights model, served locally through vLLM in FP8 with multi-token prediction, called directly with thinking switched off or on. Mistral Large 4 was called through the Mistral API as mistral-large-4-0, with reasoning effort set to none or high. The hosted default alias mistral-large-latest still pointed at the previous generation on the day we tested, so it is worth naming the model explicitly if you repeat this.

Both models got identical prompts. We ran two rounds on six short prompts (a coding task, a logic puzzle, a children's explanation in Dutch, a strict formatting task, a debugging task and a technical summary), once with thinking off and once with it on. Then we ran eight coding problems twice each with thinking on, and executed every answer against hidden unit tests. The tests were checked against our own reference solutions before the models saw a single problem.

Round one: thinking off

Without reasoning, both models answered in a few seconds and the answers were close in quality. Time to the first token favoured the local model by a wide margin (about 0.08 seconds at home, against 0.5 to 2 seconds through the API, which includes the network). Per-request decode speed was similar. Four requests at once finished in 3.6 seconds locally and 3.3 seconds through the API.

On quality, the logic puzzle was the one place Mistral was visibly crisper: it followed "be concise" better. The summary of dense versus mixture-of-experts models went the other way: Mistral claimed dense models are faster, which is backwards, while Qwen got the speed point right.

Round two: thinking on

With reasoning switched on, the gap in time grew. Both models generate at similar speed once they start, so the difference is how long they think. Mistral Large 4 spent 14,430 tokens on the six prompts against 6,355 for Qwen, and one prompt (the technical summary) took Mistral about 78 seconds. Four requests at once took 23.2 seconds locally against 56.9 seconds through the API.

Thinking helped Mistral more than Qwen. Its dense-versus-MoE summary became correct once it reasoned, where Qwen had already been correct on that point.

Round three: unit-tested code

Eight problems, each with hidden tests: duration parsing, an arithmetic expression evaluator without eval, counting longest increasing subsequences, full text justification, lexicographically smallest topological order, minimum window substring, strict Roman numerals, and a small JSON path reader. Two samples per problem per model, thinking on.

Both models scored 11 of 16, and they failed in a revealing way. Of Qwen's 5 misses, 4 were runs that used up the whole 32,000-token budget still thinking and 1 was genuinely wrong code. Of Mistral's 5, 4 hit the same budget wall and 1 was a server-side error from the API. Every answer that finished was correct except that one. So the score mostly measures which model finishes thinking in time on the two hardest problems, the expression evaluator and the JSON path reader, and on those the two were equally likely to run out.

What we would not conclude

• That one model is smarter. On the problems that finished, they were equally correct.
• Anything about long context or tool use. We did not test either. Mistral Large 4 offers a context window of about a million tokens, our local setup runs at 196K, and that gap could matter for other work.
• Anything precise. One run per prompt and two samples per coding problem. Mistral was not even consistent with itself: the same problem hit the budget limit on one attempt and passed on the next.
• A like-for-like speed comparison. The local model runs on a quiet machine next door. The API includes the network, queueing and Mistral's own load.

What we take from it

For the work we actually do, a 27B open model on our own hardware is not a compromise on code. It matches a hosted frontier model on these problems, it finishes about twice as fast once reasoning is on, and nothing leaves the building. That last point is the one our school and compliance customers care about most. The honest remaining gap is the very long context window, which this test did not touch. That is the next test worth running.

Provenance

Run on 24 October 2026. Local: Qwen3.8-27B, FP8, vLLM, thinking toggled per round, temperature 0.2 without thinking and 0.6 with it. Hosted: mistral-large-4-0 through the Mistral API, reasoning effort none or high, same temperatures. The first coding pass used a 12,000-token budget; the empty answers were rerun at 32,000, and the few that still ended early were rerun once more. The numbers on this page are computed from the saved run data, not typed in by hand.

Short version: level on code, quicker at home when thinking is on, and a small test that tells you where to look next, not who won.