Factory Bench Polyglot

Speed: solved tasks per hour

Measured over the time each model needed in its own phase, grading included. Only tasks that pass every check count as solved.

Speed and cost in detail

Costs are estimates from measured tokens, not invoices. Part of Mistral’s usage data is missing, hence the plus. Qwen runs locally; hardware and electricity are not included.

Pass rate by language

The axis starts at 85% because every value is high. These are dots, not bars.

One repository or several

Multi-repository tasks need coordinated changes in a producer and a consumer. They are harder for all three models.

Are the differences real?

Difference in pass rate per matched pair, with the 95% interval over task families. An interval that contains zero does not prove a difference.

What you should know

• Several Qwen tasks hit the per-task time limit. They count as failures, even though it is a clock limit and not a wrong solution. They occur mostly in PowerShell.
• Two Qwen tasks and one Mistral task were rejected because a protected file (policy.json) was overwritten during testing, even though the code itself may have been correct.
• Mistral ran for a while much slower because of a network problem on our side: Python first tried a broken IPv6 route. After changing the address ordering it ran normally. Mistral’s measured speed is therefore too low and cannot be compared fairly with Haiku’s.
• The tasks are synthetic and come from correlated families. PowerShell runs on Linux and Groovy uses a simulated Jenkins. This is not SWE-bench and says nothing about your own repositories.
• All models ran with the same number of workers, but that does not mean the same hardware or the same capacity at the provider.