Experiment · Coding agents

Three agents, three languages you would rather not debug, 900 trials

27 October 2026 · By Jan Janssens · Read time: 5 min · ← All experiments

We built a benchmark that hands a model a small repository, a shell and a problem in PowerShell, Groovy or Rust, then lets it work like an agent. We ran 300 tasks per model on Qwen3.8-27B (on our own GPU), Mistral Large 4 and Claude Haiku 5.5. All three solved more than 94 percent. Haiku solved the most and delivered roughly 4 to 9 times as many solved tasks per hour. Statistically, the three are closer than the headline numbers suggest.

There is also a full report page with charts, for the people who read the table first and the prose never.

The result

The gaps are 2, 3 and 1 percentage points. With 300 tasks from 90 families, the 95 percent intervals for all three paired differences include zero. So the honest reading is: all three are good at this, we cannot rank them on correctness, and the real separation is in speed and cost.

What the bench is

It is called Factory Bench Polyglot, and it is a harness that grades agents on miniature repositories. Each task ships a tiny codebase, a bug or a feature request, and some hidden tests the model never sees. The agent can list files, read, write and run commands in a sandbox with no network, and then declares itself done.

• 300 tasks, 90 families. Three or four variants of each family, 100 tasks per language: PowerShell, Groovy and Rust.
• Single and multi-repository tasks. 180 tasks touch one repository. 120 have a producer and a consumer in separate Git repositories that must stay in agreement.
• Strict grading. A task counts as solved only if every public and hidden check passes and no protected file was touched. A timeout is a failure. So is touching policy.json, even when the code was fine.
• Four workers per model. Each model ran in its own phase, 900 of 900 trials completed.

Speed and cost: this is where it separates

Haiku finished its 300 tasks in about 47 minutes. Qwen needed almost seven hours. Part of the reason is style: averaged over the final attempts, Haiku used about 6 turns and 5,000 output tokens per task, Mistral about 15 turns and 11,500 tokens, Qwen about 18 turns and 16,800 tokens. Haiku reads the problem, edits, runs the tests and leaves. The other two keep talking to themselves.

A big warning on the Mistral numbers. Halfway through, the run was crawling. The culprit was not Mistral and not the harness: Python tried a broken IPv6 route to the API first and waited about two minutes for TCP to give up, on every single request, while curl sailed past it with happy-eyeballs. One line in /etc/gai.conf and a restart fixed it, but the Mistral phase time and the dollar figure (restarted attempts get billed too) are not a fair comparison. We kept them in, with this footnote.

How they failed

Read this table twice. Qwen got the answer wrong only 6 times, the same as the others. Most of its extra failures are timeouts: on a slow task it simply ran out of clock, mostly on PowerShell. In both of Qwen's protected-file cases, the model's own test script rewrote policy.json and put it back byte-different, so the solution may well have been right. The strict rules say it fails anyway. We have not looked at Mistral's case yet.

Of the 90 families, 22 had at least one failure. Exactly one task defeated all three models: ps-multi-retention_notifications-v4, a PowerShell multi-repository task. Failures that only one model had: 14 for Qwen, 6 for Mistral, 4 for Haiku. Multi-repository tasks were harder for everyone (Qwen 91.7 percent against 96.7 on single-repository tasks), which is reassuring in a "the benchmark measures what it claims" kind of way.

What this is not

• Not SWE-bench. These are synthetic miniature repositories. They say nothing about your 400,000-line monolith with the one legendary file nobody dares to open.
• Not a ranking. The correctness differences are inside the noise, and 300 tasks from 90 correlated families is a modest sample.
• Not real Windows or real Jenkins. PowerShell runs on Linux. Groovy runs against an offline Jenkins double, so real CPS, plugins, credentials and controller restarts are not exercised.
• Not a fair hardware race. Four workers each does not mean equal hardware or equal provider capacity. One local GPU against someone else’s data centre is not a level playing field, and we did not pretend otherwise.
• Not comparable with our previous coding-agent post. Different tasks, different languages, four workers instead of eight, a different time limit. Do not put those numbers in one chart.
• Not matched in reasoning effort. We used each provider’s defaults.

What we take from it

For small, well-specified repository tasks, the local 27B model keeps up on correctness and costs nothing per token, but it is slow and times out on the hard ones. Mistral sits in the middle. Haiku is fast, cheap and, on this suite, at least as good as the others. If you are not allowed to send the code anywhere, local is a perfectly respectable answer. If you want a thousand tasks by lunch, it is not.

Provenance

Run id 20261025T061438Z-44e536, 900 of 900 trials, one repeat each. The Mistral phase was stopped and resumed after the IPv6 fix, and Haiku was added to the same run afterwards. Cost figures are estimates from the run’s own usage records, not invoices. Confidence intervals resample whole task families. All numbers come from the saved run data, and the same data drives the report page.

Short version: all three score 95 to 98 percent, Haiku is by far the fastest and cheapest, Qwen times out more than it errs, and one PowerShell task beat everybody.