🔍 Read the full analysis: Beyond Green Tests: GPT-6 Astra’s Review Of Claude’s Model Edit on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
GPT-6 Astra, running in Codex, independently reviewed a Claude Opus 5.5 experiment that edited the weights of local model Ornith 1.5. The review found reproducible problems in the test rig and corrected claims about weight changes and causation; the reported legal benchmark score remained zero. Public runs cover 10 tasks, but the reviewer did not rerun every test or inspect hidden benchmark prompts.
An OpenAI model, GPT-6 Astra, reviewed a Claude-led attempt to edit local model Ornith 1.5 and found reproducible faults in the experiment’s test rig, according to a report published by ThorstenMeyerAI.com and updated November 9. The model edit was reported to improve the system’s responses across 10 published benchmark tasks, but its legal score stayed at zero; the review focused on whether the experiment and its measurements could be trusted, not on running a competing edit.
The experiment was led by a session using Anthropic’s Claude Opus 5.5, which tried to improve Ornith 1.5’s legal-task performance through a directional weight ablation inspired by a post from MIT’s Computer Science and Artificial Intelligence Laboratory. The report says the method identified and weakened a “loop direction” in the model’s weights without additional training. On the 10 published Lexassay tasks, the edited model answered all 10, compared with four before the edit, while its legal score remained zero. The published account does not establish that the change improved legal competence.
GPT-6 Astra ran in OpenAI’s Codex and was assigned to review the implementation, not change it. Its separate check programs reproduced four problems during the first review: a resumed run could reuse answers from a different setup; an unknown task ID could produce a successful run with no evaluations; a comma conflict broke the actual command line despite passing parser unit tests; and converting weights altered values even when edit strength was set to zero. The Claude session made the fixes, the report says.
The updated account also describes a second review round. The author says all 12 of the author’s tests passed, while four of the reviewer’s six provenance checks failed. The reviewer measured a 0.51% change when one real weight matrix was stored in a different number format, with 915,360 of 1,048,576 values changing. Those figures describe format conversion, not legal quality or a general model error rate. The experiments and controls are listed publicly on Lexassay.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Independent Checks Matter
The review illustrates a practical weakness in relying on a green test suite as evidence that an AI experiment is sound. Tests can pass while real command-line use fails, and a run can appear successful even when it evaluates nothing. More seriously, reusing results from a different model or setting can make a benchmark comparison misleading. These are questions of experimental provenance, not merely software polish.
The weight-conversion finding also bears on how researchers attribute results. If storing a model in another numerical format changes its weights, then a measured difference cannot automatically be assigned to the intended edit. In this case, the report says the reviewer found that issue and the experimenter changed the storage approach. The review does not show that the edit caused a legal improvement: the reported legal score was zero, and the public task count alone does not demonstrate broader capability.
Using a reviewer from another company may offer a useful check on shared assumptions, especially when the reviewer is kept away from hidden tasks and the main implementation. But this remains one reported exercise, not a validation of either company’s models or a general finding about AI peer review. The account is also written by the experiment’s organizer, so its descriptions and measurements should be understood as the author’s report.
Popular right now
Ninja CREAMi 7-in-1 Ice Cream Maker, NC299AMZ
Shark ZU102 Rotator Pet Upright Vacuum with PowerFins HairPro & Odor Neutralizer Technology, Charcoal
Samsung 43-Inch M70H Series 4K Mini LED Smart TV (2026)
As an affiliate, we earn on qualifying purchases.
How the Two Models Were Assigned
The article says two AI sessions began working on the model experiment on November 4, 2026. Once the overlap became apparent, the work was divided. Claude retained responsibility for the experiment plan, test rig, weight edit, benchmark runs and blind grading by three Claude agents. GPT-6 Astra was instructed to review and validate the other session’s implementation.
The reviewer’s stated rules were to leave the main implementation untouched, run actual entry points rather than trust unit tests alone, and avoid the hidden benchmark material. The report says it did not inspect hidden prompts, topics, answers or reference solutions. Its preliminary files were set aside after the roles were separated, and it did not produce a second edited model. The author describes the published Lexassay tasks as the basis for the updated results.
The article was first published on November 5 and updated on November 9. The update says its figures now refer to 10 published tasks and that the runs are public. The source links to the weight-edited run, an edit control and model-card sampling, allowing readers to examine the materials described in the report.
“Review and validate the other session’s implementation.”
— The report’s account of GPT-6 Astra’s assignment
Limits of the Review Evidence
The report says the reviewer did not rerun every test during its final documentation step, was not a fourth blind grader and measured the weight-format change on a CPU rather than the production GPU path. It does not provide, in the supplied account, a full independent reproduction of every benchmark result. The relationship between the weight edit, format conversion and the change in the number of answered tasks therefore cannot be settled from the review alone.
The reported legal score stayed at zero, and the source does not establish why the edited model answered more published tasks or whether that change would hold on other benchmarks. Nor does the report establish general legal ability. The measured conversion differences apply to one matrix and should not be treated as a model-wide error rate. The account also does not specify an external audit or independent replication beyond the described Codex review.
Public Runs and Further Replication
The immediate next step for readers is to examine the public Lexassay runs, including the edited model, control and model-card sampling, and compare them with the reported fixes. The source does not announce a further benchmark, release date or planned third-party review. Any stronger conclusion about the edit’s effect would require clearly documented runs that distinguish the intended weight change from format conversion and verify that each result belongs to the stated model and settings.
For now, the development is a report of an AI reviewer uncovering flaws and prompting corrections—not evidence that Ornith 1.5 has become legally capable. The author says the reviewer did not inspect hidden tasks, but the available account leaves broader replication and performance testing open.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and measurement process for a Claude Opus 5.5-led edit of Ornith 1.5. It did not conduct a competing model-editing experiment.
Did the edit improve the model’s legal score?
No legal-score improvement was reported. The model answered 10 published tasks instead of four, but its legal score remained zero, according to the article.
What problems did the review find?
The first review reproduced four issues: stale results could be reused after a run resumed, invalid task IDs could yield empty successful runs, a real command failed despite green parser tests, and weight conversion changed values even at zero edit strength.
Were hidden benchmark tasks examined?
The report says GPT-6 Astra did not inspect hidden prompts, topics, answers or reference solutions. It reviewed the implementation and used separate checks.
What is still needed to confirm the findings?
The account does not report a full independent rerun of every test or benchmark. Further replication would help establish whether the task-count change is reproducible and separate the intended edit’s effect from changes caused by storing weights in another format.
Source: ThorstenMeyerAI.com
