🔍 Read the full analysis: FrontierCode: How An AI Pull Request Gets Judged By Maintainers on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis reported results for 23 AI models on FrontierCode, a benchmark built with open-source maintainers to assess whether coding-agent changes meet repository review criteria. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts; the results do not establish that any model can reliably produce merge-ready pull requests without human review.
Artificial Analysis published results for 23 AI models on FrontierCode, a coding benchmark that evaluates whether maintainers would accept a proposed repository change, rather than judging success by tests alone. Claude Opus 5.5 scored 52.2% and passed every blocking check in 57.7% of attempts, leaving substantial room for human review even for the top-ranked model.
Artificial Analysis added the evaluation on October 12, 2026, running each model through the same open-source agent harness, Stirrup. Each attempt was limited to 250 turns, with up to three attempts per task. The benchmark’s Main set contains 100 tasks drawn from real repositories; nine tasks were not solved by any model on any attempt, according to the source report.
GPT-6.1 Sol placed second with a 49.1% score and a 54.7% blocking-check pass rate. Claude Fable 5.1 scored 49.0%, followed by GPT-6 Astra at 47.0%. Xiaomi’s MiMo-V2.6-Pro led the open-weight models listed, with a 45.4% score. These are benchmark results, not measurements of how models perform across all software projects.
FrontierCode assigns zero if an attempt fails any blocking check. It also penalizes issues such as excessive changes or edits outside the relevant files, while quality checks affect scores for attempts that clear the blockers. The review stage uses GPT-6 Luna (medium) as an LLM judge, and the source says attempts flagged by Cognition’s trajectory scanner for solution lookup also receive zero.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
Many coding evaluations focus on whether a solution passes tests. FrontierCode also asks whether a patch stays within scope, avoids regressions, includes useful tests and meets review criteria. That makes its results relevant to teams considering AI coding agents: a change can be technically functional yet still create work for maintainers or fall short of repository standards.
The top model’s 57.7% blocking-check pass rate is a benchmark result, not proof of readiness for unsupervised use. It indicates that even under these test conditions, more than four in ten attempts did not clear every required blocking check. The finding supports a measured conclusion: review remains part of the process, and the leaderboard alone does not show that agents can safely merge changes without people.
The scores also differ from general intelligence rankings cited in the source. Claude Sonnet 5.5 ranks fifth on FrontierCode despite a stronger position on Artificial Analysis’ Intelligence Index, while DeepSeek V4.1 Flash ranks higher on the coding benchmark than its general score might suggest. Teams selecting models for repository work may find task-specific evaluations more informative than broad capability rankings.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Pull Requests
Cognition and open-source maintainers developed FrontierCode. The benchmark was announced in June 2026, and version 1.1 followed in July with rules intended to prevent agents from looking up solutions online. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The full benchmark has 150 tasks; Artificial Analysis evaluated the 100-task Main set. The source describes the tasks as private, limiting the chance that models encountered the solutions during training.
Evaluation spans six areas: hidden tests for correctness; command checks for regressions and cleanliness; reverse tests to check that an agent’s tests fail on the original code; adaptive tests for complex behavior; limits on files and diff size; and an LLM code review against maintainer criteria. Checks can be blocking or non-blocking. Failing a blocking check results in zero, while passing all blockers yields a score between 70% and 100%, depending on quality checks, according to the benchmark description.
Artificial Analysis’ reported cost figures show a wide spread. GPT-6 Luna (max) scored 41.9% at $0.09 per task, MiMo-V2.6-Pro scored 45.4% at $0.18, GPT-6.1 Sol scored 49.1% at $0.81, and Claude Opus 5.5 scored 52.2% at $10.15. Those figures are the source’s per-task estimates. Moving from Sol to Opus adds 3.1 score points at about 12.5 times the cost, illustrating the trade-off in the reported results rather than a universal price for using either model.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results cover 23 models and the 100-task Main set, not every model or the full 150-task benchmark. As of October 13, the source report listed no FrontierCode result for Gemini 4 Argon, Muse Wonder, Muse Wonder Full or Mistral Large 4 Preview. Their eventual performance is unknown, so the current ranking may change as more results are added.
The available material does not establish how closely benchmark scores predict outcomes in a particular company’s repositories, workflows or review process. It also does not provide enough information here to independently assess the reliability of the LLM review judge or the cost estimates across different usage patterns. The results should be read as performance under FrontierCode’s stated setup, not as a guarantee about production code.
As an affiliate, we earn on qualifying purchases.
More Models and Evaluations Ahead
The immediate next development is whether Artificial Analysis publishes results for the models that were missing from its October 13 snapshot, and whether later evaluations expand coverage beyond the 100-task Main set. Any added scores would make the comparison more complete, but would not remove the need to check how well the benchmark represents individual projects.
For maintainers and engineering teams, the practical next step is to treat these results as one input when comparing coding agents. The reported ranking captures correctness, scope and review-related checks under a common harness; teams still need to judge whether an agent’s proposed changes fit their own codebase and standards. No deployment or adoption decision follows automatically from a leaderboard position.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated code changes against repository tasks using tests, regression checks, scope limits and an LLM-based code review. Its stated goal is to assess whether a change would meet maintainer-style acceptance criteria, not just whether tests pass.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led the reported results with a 52.2% FrontierCode score. It passed all blocking checks in 57.7% of attempts.
Does the leading score mean AI pull requests are ready to merge automatically?
No. The benchmark reports performance on a particular task set and setup. The top model did not clear every blocking check in all attempts, and the results do not establish that AI-generated changes can be merged without human review.
How many tasks and models were tested?
Artificial Analysis tested 23 models on the 100-task Main set. FrontierCode’s full benchmark contains 150 tasks, according to the source report.
Which models were missing from the results?
As of October 13, 2026, the source report listed no FrontierCode result for Gemini 4 Argon, Muse Wonder, Muse Wonder Full or Mistral Large 4 Preview. The reason for their absence and when results may appear were not specified.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
