🔍 Read the full analysis: FrontierCode: What Maintainers Look For In AI-Generated Code on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added FrontierCode results on October 12, 2026, after testing 23 AI models on repository tasks designed with open-source maintainers. Claude Opus 5.5 led with a 52.2% overall score and cleared every blocking check in 57.7% of attempts; costs and results varied widely, and nine tasks were unsolved by every model.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark designed with open-source maintainers to judge whether a proposed code change is suitable to merge, rather than simply whether it passes tests. Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in only 57.7% of attempts, indicating that even the top result leaves substantial room for human review.
The results, added to Artificial Analysis’s evaluations on October 12, 2026, cover 100 tasks from FrontierCode’s Main set. Opus 5.5, using the max configuration with fallback, ranked first. GPT-6.1 Sol scored 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Nine models from six labs scored between 43% and 52%; results fell more sharply below that group, with Grok 4.7 at 33.8% and Nemotron 3 Ultra at 10.5%.
The overall score is not the same as the pass rate. FrontierCode gives an attempt zero if it fails any blocking check; attempts that clear those checks receive a score from 70% to 100%, based on additional quality criteria. Opus cleared all blocking checks in 57.7% of attempts, while GPT-6.1 Sol did so in 54.7%. Nine of the 100 tasks were not solved by any model on any attempt, according to the source report.
Artificial Analysis ran each model through the same open-source agent harness, Stirrup, allowing up to 250 turns per attempt and three attempts per task. A review stage used GPT-6 Luna (medium) as the judge. Those testing choices matter when interpreting the ranking: the figures report performance in this setup, not a guarantee of how a model will perform in every development workflow.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Merge-Ready Scores Matter
FrontierCode focuses on a gap between working code and reviewable code. A patch can pass tests yet still add unnecessary files, exceed the intended scope, introduce weak tests or fail to meet maintainers’ standards. By checking these factors alongside correctness, the benchmark aims to measure a task developers face before approving an AI-generated change: whether the patch is acceptable in the repository.
The results suggest teams should not treat a high benchmark rank as a reason to remove review. Even the leading model failed to clear a blocking check in more than four out of every ten attempts in this evaluation. That figure is specific to the benchmark and should not be read as a general failure rate across all code tasks, but it underscores that human review remains relevant for the measured work.
Cost also changes the practical comparison. Artificial Analysis put the cost per task at $10.15 for Opus 5.5 and $0.81 for GPT-6.1 Sol, while Sol’s score was 3.1 percentage points lower. The source report identifies GPT-6 Luna, MiMo-V2.6-Pro, Sol and Opus as points on its cost frontier, where no other evaluated model is both cheaper and better. Buyers therefore face a trade-off between score and expense, not a single uncontested best choice.
static code analysis tools for developers
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was created by Cognition and open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against searching online for task solutions. More than 20 maintainers developed tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. The full benchmark contains 150 tasks; Artificial Analysis ran the 100-task Main set. The tasks are private, according to the source, to limit the chance that models encountered them during training.
Each patch is assessed on six areas: hidden tests, command checks, tests that must fail against the original code, adaptive tests for complex behavior, limits on files and diff size, and an LLM code review against maintainer criteria. Failing a blocking check means an attempt scores zero. The benchmark also penalizes excessive changes, edits outside relevant files and unhelpful tests. Cognition’s trajectory scanner can assign a zero if it flags an attempt for looking up a solution.
The leaderboard also differs from general intelligence rankings. In the source report’s comparison, Claude Sonnet 5.5 ranked fifth on FrontierCode despite ranking second on the cited Intelligence Index among the models compared. DeepSeek V4.1 Flash placed seventh on FrontierCode. This suggests broad capability scores alone may not predict performance on repository changes graded for maintainability; it does not establish how models will rank on every coding benchmark.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The published evaluation covers 100 tasks and 23 models, not every task in the 150-task benchmark or every available coding model. The source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of October 13. Their absence means the ranking cannot yet compare those systems on this benchmark.
The source does not provide enough detail here to establish how closely these task results predict performance across different repositories, programming languages or teams. Nor does a benchmark pass rate state how often a model would produce an acceptable change in a company’s own workflow. Results depend on the task set, agent configuration, attempt limits and review procedure. Real-world performance and costs may differ, and the source does not report an independent replication of this evaluation.
As an affiliate, we earn on qualifying purchases.
Models and Tasks Still to Come
The next useful comparison will depend on results for models missing from the October 13 leaderboard and on whether later evaluations expand beyond the 100-task Main set. The source report does not give dates for new results or a planned update schedule. Readers should treat any later ranking as a fresh snapshot and check which models, configurations, tasks and costs it includes.
For teams assessing these systems now, the reported figures offer a benchmark-based comparison, not a substitute for testing their own repositories. The open question is whether future results raise blocking-check pass rates while keeping costs manageable—and whether performance holds across a broader range of codebases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated changes to real repository tasks using hidden and adaptive tests, command checks, scope limits and an LLM review based on maintainer criteria. A failed blocking check gives the attempt a zero.
Which model ranked first?
Claude Opus 5.5 (max with fallback) had the highest reported overall score, 52.2%. It cleared all blocking checks in 57.7% of attempts.
Does a 52.2% score mean the model passed 52.2% of tasks?
No. The overall score and blocking-check pass rate are different measures. Opus scored 52.2% overall and cleared all blocking checks in 57.7% of attempts; attempts that clear every blocking check receive a score between 70% and 100% based on other quality criteria.
How much did the leading models cost per task?
Artificial Analysis reported a cost of $10.15 per task for Opus 5.5 and $0.81 for GPT-6.1 Sol. The source’s cost figures apply to its evaluation setup and may not match other use.
Were all major coding models included?
No. As of October 13, the source report listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
