🔍 Read the full analysis: The Second AI Review That Flagged Bugs In Claude’s Model Edit on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A review by OpenAI’s GPT-6 Astra found reproducible bugs in a Claude-led experiment that edited the weights of the local model Ornith 1.5. In the second review round, the author’s 12 tests passed, but four of the reviewer’s six provenance checks failed; the reviewer also corrected claims about weight changes and what the experiment showed.
A review by OpenAI’s GPT-6 Astra found reproducible bugs and overstatements in a Claude Opus 5.5-led experiment that edited the weights of local model Ornith 1.5. The review did not run a competing model-improvement experiment; it checked the implementation and claims, finding that green tests had missed faults that could affect the benchmark results.
The experiment, conducted by the author’s Claude session, used a directional ablation to weaken a “loop direction” in Ornith 1.5’s weights without training. According to the source article, the edited model answered all 10 published benchmark tasks, compared with four for the original, while its legal score remained zero. The runs are publicly available on Lexassay. Those results are the experiment’s reported measurements, not evidence that the model gained legal capability.
GPT-6 Astra was assigned to review the implementation, not alter it. The reviewer wrote separate checks, did not inspect hidden benchmark prompts or answers, and left fixes to the Claude session. Its first review identified four faults: resuming could reuse results from a different setup; a mistyped task ID could produce a successful run with no evaluations; the real command line failed despite passing parser unit tests; and converting weights changed them even when the edit strength was zero.
The review measured a 0.84% change in one matrix during the original zero-strength conversion, with 105,315 of 1,048,576 reconstructed values differing. The author says the implementation was changed to reject unknown IDs, use slash-separated settings and store edited tensors in higher-precision bf16 format. In the second review round, all 12 of the author’s tests passed, while four of the reviewer’s six provenance checks failed. The source says these measurements concern code and weight conversion, not legal accuracy or a general error rate.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Benchmark Provenance Matters
The review highlights a problem that can be missed when a model-editing experiment is judged mainly by its output scores: the benchmark pipeline must reliably record what was tested. If a resumed run silently carries over answers from another model or configuration, or if an invalid task ID creates an empty but successful run, results may appear complete without representing the intended test.
The weight-conversion finding also complicates attribution. When a nominally zero-strength edit still alters values, a result cannot automatically be assigned to the proposed edit alone. That does not show that the published benchmark outcome was caused by conversion; it shows why controls and accurate descriptions of the data path matter. Publishing runnable tests and measurements gives readers a way to inspect those claims, while the remaining review limits mean the audit is not a complete validation of every result.
Products Worth Considering
Jackery Solar Generator 1000 V2 with 100W Solar Panel, 1024Wh LiFePO4 Battery
LG UltraGear 32GX850A-B 32-inch 4K UHD OLED Gaming Monitor
Amazon Kindle Paperwhite (2024, 16 GB) – 7-inch Glare-Free Display, Black
As an affiliate, we earn on qualifying purchases.
How the Two-Model Review Was Set Up
The work began on November 4, 2026, with two AI sessions working on the same broader project. The author says they were then given different roles: Claude Opus 5.5 retained responsibility for the experiment, including the edit, benchmark runs and blind grading by three Claude agents; GPT-6 Astra, running in Codex, was assigned to review the implementation. Preparatory files from Astra’s earlier work were set aside, and the review produced no second edited model.
The article frames the arrangement as an independent check because the reviewer came from a different company and did not change the main implementation. It also says the reviewer did not read hidden benchmark tasks. However, independence here describes the role assignment and separation of work; it does not mean the reviewer repeated the entire experiment or independently verified every stage. The article’s November 9 update says the figures refer to 10 published Lexassay tasks and that the runs are public.
What the Review Did Not Verify
The source article says the reviewer did not rerun every test during its final documentation step, did not serve as a fourth blind grader, and measured the conversion on a CPU rather than the production GPU path. It therefore does not establish that every benchmark result is independently reproduced or that CPU measurements match production behavior.
The article also does not provide enough detail to determine the precise cause or benchmark impact of each provenance-check failure, or to establish how much of the reported 10-task result, if any, was affected by the identified faults. The author says the fixes were tested, but a passed test suite is not itself confirmation of legal competence: the reported legal score remained zero. The review’s findings should be read as an audit of implementation and claims, not as proof that the model edit works or fails more broadly.
Public Runs and Further Validation
The next step available to readers is to inspect the published Lexassay runs, including the weight-edited model, edit control and model-card sampling, and compare the recorded results with the revised implementation. The source says the Claude session made the fixes and added tests that fail on the old code. It does not announce a further review date or a new benchmark milestone.
Any stronger conclusion would require additional validation, including checks across the production GPU path and a clear account of the provenance failures and their possible effects. Until that information is available, the confirmed development is that a second AI review found faults, prompted code and wording changes, and left questions about the scope of the audit unresolved.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and claims in a Claude-led experiment on Ornith 1.5. The source says it did not run a competing model-improvement experiment or inspect hidden benchmark tasks.
What bugs did the review find?
The first review found that a resumed run could reuse results from another setup, an unknown task ID could yield a successful run with zero evaluations, the real command line could fail despite green parser tests, and weight conversion changed values at zero edit strength.
Did the edited model show better legal performance?
The source reports that the edited model answered all 10 published tasks, compared with four for the original, but says the legal score stayed at zero. The reported task count does not establish improved legal capability.
Were all results independently reproduced?
No. The source says the reviewer did not rerun every test in its final documentation step, was not a fourth grader and measured on a CPU rather than the production GPU path. It describes a review, not a full independent reproduction of the experiment.
Source: ThorstenMeyerAI.com
