🔍 Read the full analysis: 6 Insights From Our Home AI Weight-Space Editing Experiment on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A home experiment used directional ablation to reduce looping in the local AI model Ornith 1.5 35B-A3B. On the same setup, it completed all 10 published Lexassay tasks, up from 4, but its legal score stayed at zero. The experiment was small, and its authors say the edit improved completion, not legal knowledge.
A home experiment found that a direct edit to the weights of local AI model Ornith 1.5 35B-A3B helped it finish more legal benchmark questions, but did not improve its legal score. On the same test setup, the edited model answered 10 of 10 published tasks, compared with 4 of 10 before the edit; both versions scored zero on the benchmark’s legal evaluation.
The experiment was inspired by an MIT CSAIL report on “weight-space programming,” but used a different, established method called directional ablation. According to Thorsten Meyer’s account, a Claude Opus 5.5 session wrote code to run Ornith, identify a direction associated with looping, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. The process did not train the model, add legal data or have an AI read the weights as text.
On the 10 published Lexassay tasks, the edited version produced answers to all 10, while the compared run on the same setup produced answers to four. The edited answers met 7 of 50 checked criteria, up from 3, but the legal score remained zero. The account reports 55 wrong or invented citations among the edited answers. The benchmark checks legal citations against primary sources and runs without internet access or tools.
The unchanged model also answered 9 of 10 tasks when run with the sampling settings recommended in its model card, without any weight edit. Meyer says this result makes the sampling configuration relevant to interpreting the apparent gain. The article’s updated figures cover the 10 published tasks, and it says the runs, answers, grades and citation checks are public on Lexassay.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completed Answers, Same Legal Score
The result separates two problems that can look similar in an AI transcript: a model may fail to provide an answer because it loops, or because it lacks the knowledge to answer correctly. In this experiment, the edit was associated with more completed responses, but not with a higher legal score. That distinction matters for anyone evaluating model improvements: producing a finished answer is not evidence that the answer is accurate.
The experiment also cautions against attributing gains to a weight edit without checking ordinary settings. The unchanged model answered nine tasks under its recommended sampling settings, close to the edited model’s 10 on the reported comparison. The work therefore offers a limited test of a technique, not evidence that editing weights can make a general-purpose model reliable for legal work. The article explicitly says the experiment is not legal advice.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Clicky Switches
DJI Mini 4 Pro Fly More Combo with DJI RC 2
Anker SOLIX C1000 Gen 2 Portable Power Station with 200W Solar Panel
As an affiliate, we earn on qualifying purchases.
From MIT’s Edit to a Local Test
The experiment followed a report by MIT CSAIL researchers Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Their post describes a method they call weight-space programming: a general-purpose model reasons about another model’s weights and edits them directly, without a training loop. In a Qwen example, the researchers reported removing a looping behavior and raising a mathematics score from 7.9% to 48.1%. Those are the source article’s account of the MIT result, not findings from the home experiment.
Meyer’s test instead used a familiar activation-based approach: run the target model, measure internal signals and project out a selected direction. Ornith was chosen because, on the benchmark, it showed repeated text and failed to finish two published tasks within a 62,000-token output limit. The account says other candidate models failed in different ways, including lacking relevant knowledge, while another looping model was too large for the available machine and used a different token budget.
Lexassay asks models 15 questions covering German, US and EU law. The source says 10 tasks are published and five remain hidden to reduce the possibility of tuning to the test. The figures in this report apply to the published tasks, not the full set.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, describing the experiment’s finding
Small Test, Uneven Comparisons
The findings are limited by one run per setting, a small development set and a loop detector that changed during the experiment, according to Meyer. The article also says three graders came from the same model family. Although OpenAI’s GPT-6 Astra, run in Codex, independently reviewed the code, numbers and claims and found faults, this was one reviewer—not a second experiment or a replication.
The two unchanged-model results are not directly comparable: the original published run answered eight of 10 tasks, while the run on the separate setup used for comparison answered four. The recommended-sampling run answered nine. The account does not establish how much of the difference between these outcomes came from the weight edit, sampling settings or setup differences. Nor does this single experiment show whether the effect would hold across models, tasks or repeat runs.
Public Runs Allow Further Checking
The source says the experiment’s runs are public on Lexassay, including the original run, the model-card sampling run, the weight-edited run and an edit control. The method page describes how the edit was performed. Readers can inspect the answers, grades and citation checks behind the reported figures.
No follow-up experiment or replication is specified in the source material. The next useful evidence would include repeated runs under matched settings, a stable loop-detection method and evaluation on additional tasks, including the five hidden Lexassay questions. Until then, the reported result supports a narrow conclusion: this edit helped one local model complete more tasks in one comparison, while its legal score remained zero.
Key Questions
What did the weight edit change?
The experiment removed a measured looping direction at half strength from the output weights of 27 of Ornith’s 40 layers. It used directional ablation, not the MIT method of having an AI reason directly over weights as text.
Did the edit improve the model’s legal accuracy?
Not according to the reported legal score: it remained zero. The edited answers met 7 of 50 checked criteria, compared with 3 before, and included 55 wrong or invented citations, according to the source article.
How many benchmark tasks did the edited model answer?
It answered 10 of the 10 published Lexassay tasks in the reported same-setup comparison, compared with four before the edit. These figures concern the published tasks, not all 15 benchmark questions.
Was the experiment a replication of the MIT work?
No. It was inspired by the MIT report but used a different technique and tested one local model. The source describes it as a related, simpler experiment, not a replication of the researchers’ weight-space programming method.
What are the main limitations?
The report cites one run per setting, a small development set, changes to the loop detector, graders from the same model family and comparisons across local setups that were not directly comparable. The independent review checked the work but was not a second experiment.
Source: ThorstenMeyerAI.com
