🔍 Read the full analysis: FrontierCode: Would A Maintainer Trust Your AI’s Code Changes? on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Artificial Analysis published results from 23 AI models tested on FrontierCode, a coding benchmark that checks whether maintainers would accept a code change, not just whether tests pass. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts; the results do not establish that any model can reliably produce merge-ready changes without human review.
Artificial Analysis published results from 23 AI models tested on FrontierCode, a benchmark designed to judge whether a code change meets repository maintainers’ standards rather than merely passing tests. Claude Opus 5.5 ranked first with a 52.2% score, but passed every blocking check in 57.7% of attempts, leaving substantial room for human review.
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026. The evaluation used an open-source agent harness called Stirrup, with a limit of 250 turns per attempt and up to three attempts per task. The reported results cover the benchmark’s 100-task Main set; the full benchmark contains 150 tasks.
Claude Opus 5.5 (max with fallback) scored 52.2%, ahead of GPT-6.1 Sol (max) at 49.1% and Claude Fable 5.1 (max with fallback) at 49.0%. The pass rate is a separate measure: Opus cleared all blocking checks in 57.7% of attempts. Nine of the 100 tasks were not solved by any tested model on any attempt.
The benchmark checks patches across areas including hidden tests, command-line checks, whether an agent’s tests fail on the original code, limits on the size and location of changes, and an AI review against maintainer criteria. A failure on any blocking check gives an attempt a zero. Artificial Analysis says GPT-6 Luna (medium) served as the review judge. The benchmark also assigns zero to attempts flagged by Cognition’s trajectory scanner for looking up a solution.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Tests Is Not Enough
FrontierCode tests a point that conventional coding benchmarks can miss: a patch may pass tests yet still be too broad, poorly tested or otherwise unsuitable for a project’s maintainers. Its scoring makes those review concerns part of the evaluation. The results give developers and organizations a more repository-focused comparison than a leaderboard based only on task completion or test success.
The leading result also sets a limit on claims about autonomous coding. Opus passed every blocking check in 57.7% of attempts, not all of them, and the benchmark’s score is not evidence that a model can independently manage production changes. Results suggest that review remains a necessary part of using these systems. They do not, by themselves, measure performance across every project or real-world workflow.
Cost is another practical consideration. Artificial Analysis reports a cost of $10.15 per task for Opus, compared with $0.81 for GPT-6.1 Sol, which scored 49.1%. That is a 3.1-point score difference for a 12.5-times-higher listed cost per task. Such figures can help teams compare benchmark performance with expense, but do not show the total cost or value of deploying a model in a particular organization.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Judges Patches
FrontierCode was developed by Cognition and open-source maintainers and announced in June 2026, according to the source report. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. The report says maintainers spent more than 40 hours on each task. The tasks are private, and version 1.1, released in July, added rules against looking up solutions online.
Unlike a simple test-run score, FrontierCode combines correctness checks with limits on scope and an LLM-based code review. Passing the blocking checks earns a score between 70% and 100%, depending on quality checks; failing any blocking check results in zero. According to the report, oversized diffs, edits to unrelated files and unhelpful tests can reduce scores.
The top nine models scored between 43.2% and 52.2%. Among the listed results, Xiaomi’s MiMo-V2.6-Pro scored 45.4%, while DeepSeek V4.1 Flash scored 44.0% and Kimi K3 scored 43.6%. The source says Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode results as of October 13, so the table does not represent every notable model.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, quoted in Cognition’s announcement
What the Results Cannot Show
The published results do not establish how well the models would perform across all repositories, programming languages or development teams. The tasks come from a defined, private set of 100 cases in the Main evaluation, and the source does not provide enough information here to assess how representative those cases are of everyday software work. Private tasks reduce the chance that models saw the exact solutions during training, but do not settle broader questions about benchmark coverage.
It is also unclear from the reported figures how closely the AI review judge’s decisions match the views of human maintainers in practice. The scores depend on benchmark rules, the agent harness, attempt limits and review setup. Several models—including Gemini 4 Argon and Muse Wonder Full—had no result in the reported table as of October 13, meaning the rankings could change as additional evaluations are published.
More Models May Change the Rankings
The immediate next point to watch is whether Artificial Analysis publishes results for models missing from the October 13 comparison. The source identifies Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as lacking FrontierCode scores at that time, but gives no release schedule for further tests.
For readers comparing coding tools, the current results provide a benchmark snapshot rather than a final verdict. Further model entries, repeated evaluations or evidence from maintainers using these systems in actual repositories would help show how well FrontierCode scores translate into day-to-day review and merge decisions.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates code changes against tests, scope limits and review criteria, rather than treating test passage alone as proof that a patch is acceptable.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s reported results, scoring 52.2%. It passed every blocking check in 57.7% of attempts.
Does the result mean AI can submit code without human review?
No. The leading model failed to clear every blocking check in a substantial share of attempts, and the benchmark does not establish that AI-generated patches are safe to merge without review in real projects.
How many tasks and models were included?
Artificial Analysis tested 23 models on the 100-task Main set. The full FrontierCode benchmark contains 150 tasks, according to the source report.
Which prominent models were missing from the results?
As of October 13, 2026, the source report listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
