🔍 Read the full analysis: Why GPT-6 Astra’s Review Found Issues In Claude’s Model Edit on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A review by OpenAI’s GPT-6 Astra found reproducible flaws in the benchmark workflow for an experiment led by Anthropic’s Claude Opus 5.5. The reported problems included stale results being reused, invalid task IDs producing empty runs, a broken command-line path, and weight changes even at zero edit strength. The review also narrowed claims about the experiment; it did not independently run a competing model-editing test.
OpenAI’s GPT-6 Astra, running in Codex, found reproducible problems in the benchmark and weight-conversion workflow for an experiment led by Anthropic’s Claude Opus 5.5, according to a Nov. 5 account updated Nov. 9. The review matters because some faults could have made old or incomplete results look like valid measurements, even while software tests passed.
The experiment attempted to improve the local model Ornith 1.5 for legal tasks by changing its weights, without additional training. The source account says the edited model answered all 10 published benchmark tasks, compared with four for the original, while its legal score remained zero. Runs and related materials are said to be public on Lexassay.com. Those results are reported by the experiment’s author; the review described here checked the implementation and claims rather than conducting a second model-improvement experiment.
In its first review round on Nov. 4, GPT-6 Astra reported four reproducible faults. A resumed run could keep an answer generated under different model or sampling settings; an unknown task ID could finish as a successful run with no evaluations; a real command-line invocation failed despite green parser unit tests; and converting weights changed values even when the requested edit strength was zero. The author says the experiment team corrected each issue and added tests designed to fail on the earlier code.
The later review tested the revised work. The account says all 12 of the author’s tests passed, while four of the reviewer’s six provenance checks failed. The reviewer also measured one matrix conversion in a separate format: it changed 915,360 of 1,048,576 values, a reported 0.51% movement. That figure describes the conversion measurement, not legal performance or an error rate. The source also reports an earlier zero-strength conversion measurement of 0.84% and 105,315 differing values, before the team’s fix.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Benchmark Provenance Matters
The review shows why a passing test suite alone may not establish that a benchmark result reflects the intended experiment. If a run can silently reuse answers from different settings, accept an invalid task as complete, or fail on the actual command line, the resulting record can be misleading even without any deliberate manipulation. These are reproducibility and provenance risks: they affect whether readers can tell what was tested and how.
The weight-conversion finding also bears on causal interpretation. If weights change during format conversion even when edit strength is zero, an observed outcome cannot automatically be attributed to the intended directional edit. The author says the team changed how edited tensors were stored after the review. That correction addresses a potential confound in the procedure, but it does not by itself establish that the edit improved legal reasoning: the reported legal score stayed at zero.
For readers comparing AI systems, the episode is a case of one model being used as a technical reviewer of another model’s work. The account describes GPT-6 Astra as identifying implementation faults and challenging overstrong wording, not as validating the headline performance result. That distinction matters when interpreting AI-generated reviews and benchmark claims.
Products Worth Considering
Anker SOLIX C1000 Gen 2 Portable Power Station with 200W Solar Panel
Amazon Kindle Paperwhite (12th Generation, 2024), 16GB, 7-Inch Display, Raspberry
Bose QuietComfort Ultra Wireless Over-Ear Headphones with Spatial Audio and Noise Cancellation, Black
As an affiliate, we earn on qualifying purchases.
How the Two AI Roles Split
The source says two AI sessions began working on the experiment on Nov. 4. After the overlap became apparent, Claude Opus 5.5 kept responsibility for the experiment: its plan, test rig, weight edit, benchmark runs, and blind grading by three Claude agents. GPT-6 Astra was assigned a separate review role and was instructed not to alter the main implementation.
The reviewer reportedly wrote separate checking programs, exercised real entry points with small reproducible examples, and did not inspect hidden benchmark prompts, answers, topics, or reference solutions. The source says early preparatory files from GPT-6 Astra were set aside after the roles were split, and no second edited model resulted. The account’s update dated Nov. 9 says its figures refer to the 10 published tasks and that runs are public on Lexassay.
The review also revised how the experiment was described. According to the author’s account, GPT-6 Astra objected to saying the zero-strength result was “exactly identical,” said an average of wrong citations per answer needed a qualifier, and found “parallel Codex experiment” too broad. It also judged a causal claim based on one control stronger than the evidence supported. These are reported corrections from the review, not independent confirmation of the benchmark’s legal conclusions.
“Review and validate the other session’s implementation.”
— GPT-6 Astra’s review instructions, as reported by the author
Limits of the Review Evidence
The account is a report by the experiment’s author, and the underlying runs are described as public, but this article has no separate independent audit of those files. The source says GPT-6 Astra did not rerun every test during its final documentation step, was not a fourth grader, and measured on a CPU rather than the production GPU path. Its review therefore does not amount to a complete replication of the experiment.
It is also not clear from the supplied material how the 10-task benchmark was selected, how broadly its results generalize, or whether the reported difference in answered tasks would persist under other settings or on a larger evaluation. The source gives the legal score as zero but does not provide enough detail here to independently assess the scoring method. The benchmark figures and the reviewer’s measurements should be read within those limits.
Public Runs and Follow-Up Checks
The source says the experiment runs, weight-edited model, edit control, and model-card sampling materials are available on Lexassay.com. Those public artifacts allow other researchers to inspect the disclosed procedure and measurements, though the supplied account does not announce a scheduled independent replication or further review.
The next useful checks would be whether outside reviewers can reproduce the corrected workflow, confirm that the revised storage path avoids the earlier conversion effect, and test the result across more tasks and settings. Until then, the reported change in how many tasks received answers should not be treated as evidence of improved legal accuracy, especially given the stated zero legal score.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and reporting for a Claude Opus 5.5-led weight-editing experiment on Ornith 1.5. The source says it did not run a competing model-improvement experiment.
What problems did the reviewer find?
The first review reported that resumed runs could reuse answers from different settings, unknown task IDs could yield empty successful runs, a command-line path failed despite passing unit tests, and weight conversion changed values at zero edit strength.
Did the edit improve the model’s legal score?
Not according to the source account: it says the edited model answered all 10 published tasks, compared with four for the original, but the legal score remained zero.
Were all of GPT-6 Astra’s checks successful?
No. The account says all 12 of the author’s tests passed in the second review round, while four of the reviewer’s six provenance checks failed. It also says the reviewer did not rerun every test during final documentation.
Can the findings be independently checked?
The author says runs and related experiment materials are public on Lexassay.com. The supplied account does not report an independent outside replication, so public availability should not be confused with independent confirmation.
Source: ThorstenMeyerAI.com
