🔍 Read the full analysis: 71 AI Configurations, One News Pipeline, And A Surprising Trade-Off on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Thorsten Meyer says a test of 71 AI configurations for English- and German-language news writing found GPT-6 Astra scored highest, while GPT-6 Luna delivered lower but consistent scores at about one-hundredth the price. The test also found no reliable benefit from higher reasoning settings and identified two fallback models that produced fabricated quotes or unusable output.
Thorsten Meyer says he tested 71 AI model configurations on a production news-writing task and found that GPT-6 Astra produced the strongest articles, while GPT-6 Luna was his lower-cost choice for routine publishing. The test also found that higher reasoning settings did not deliver a reliable quality improvement, and that two models used as backups generated fabricated quotes or failed to produce usable articles.
In a report published October 13 on ThorstenMeyerAI.com, Meyer describes testing models from September 8 through October 13 with the same prompt and news sources used in his production pipeline. The configurations covered API and subscription models, free-tier options accessed through OpenRouter and open-weight models running on his own hardware. Each source was turned into an article in English and German, then assessed by two AI judges.
GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test and received a source-fidelity score of 5.00 there. In a separate 12-article robustness run, it averaged 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer put Astra’s cost at about $0.12 per article and its writing time at roughly 50 seconds. GPT-6 Luna averaged 3.67 across the robustness run, with source fidelity of 4.71 and valid output for every article; its reported production cost was about $0.0013 per article and median time was 17 seconds.
The reasoning-effort comparison covered six GPT-6 Luna settings. Scores varied across English and German, but Meyer said the differences were within the test’s noise, including a small apparent advantage for the high setting in the larger run. The maximum setting cost about 12 times as much and took about 12 times as long as the no-effort setting, without a reliable score gain. Meyer says he kept the no-effort configuration in production.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Cost and Reliability in Production
The results illustrate why a model leaderboard may not answer a publisher’s operational question. The highest-scoring system in this test, GPT-6 Astra, cost roughly 100 times as much per article as Luna, while Luna still achieved a consistent score and valid outputs in the robustness run. For a high-volume workflow, that difference could shape whether the strongest model is used on every story, reserved for selected work or not used at all.
The reported backup failures also show that quality is not only a matter of the primary model. Meyer says GPT-4.1 nano scored 1.62 in English and 1.68 in German, with source fidelity of 1.00, and that the judges flagged invented quotations. A locally run gpt-oss-120b produced no valid articles in four attempts, according to the report. A fallback that writes unsupported material or breaks the output format can undermine the reliability of the overall pipeline, even if it is invoked only when the main route fails.
As an affiliate, we earn on qualifying purchases.
How the News Pipeline Was Tested
Meyer’s evaluation focused on a narrow task: turning a supplied news source into a finished article using the exact prompt used by his publishing pipeline. It used two sources, one described as a 6,340-character product filing and another as a 619-character political news item. Meyer said the shorter source was especially relevant because many of his production articles begin with limited material.
Two AI judges, GPT-6.1 Sol using high reasoning and Claude Opus 5.5, saw the original source and scored articles from one to five for source fidelity, fabrication, clarity about uncertainty, news value, structure and polish. German-language output was also assessed for language quality. The strongest configurations then took part in a robustness run of 10 additional articles, mostly based on thin sources. Meyer said an article counted only if it produced valid output after the same repair step used in production. He cautioned that, with just two articles in the initial test, score gaps under about 0.3 points could be noise; he treated the larger robustness run as more dependable for ranking.
The comparison also included models with different reputations on other tasks. Meyer reports that Claude Opus 5.5 scored 3.46 in English and 3.39 in German, with source fidelity of 3.75, and averaged 3.52 in the robustness run. Claude Sonnet 5.5 scored 3.67 and 3.61 in the two languages, which Meyer described as effectively tied with Luna, while costing about $0.036 per article on an API-equivalent basis. These are results from Meyer’s specific test, not a general ranking of models across all uses.
“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”
— Thorsten Meyer, in his October 13 report
Limits of the Model Comparison
The findings come from one publisher’s pipeline, one production prompt and a small set of source material. The report describes two articles in the standard test and 10 additional articles in the robustness run for the strongest configurations; it does not establish how the models would perform across a larger range of outlets, topics, prompts or editorial standards. Meyer’s own caution about small score differences also means near-ties should not be treated as definitive rankings.
The results were graded by AI judges rather than described as an independent human editorial review. Although both judges saw the source, the report does not establish how their scoring compares with human editors’ judgments or whether the same findings would hold in other languages. The article also does not provide the full results for all 71 configurations in the supplied material. The scale of any real-world risk from fabricated fallback quotes is not quantified beyond the examples and test outcomes Meyer reports.
The report says GPT-6 Luna now runs first in the news route after the two backup models were removed on October 7, but its description of the current failover setup ends mid-sentence. It is consequently unclear from the supplied account what model, if any, currently serves as Luna’s replacement fallback.
Production Route After the Test
Meyer says the production pipeline remains on GPT-6 Luna with reasoning effort set to none, while GPT-4.1 nano and gpt-oss-120b were removed from the news route on October 7. The published account does not specify the replacement arrangement for failed or unavailable Luna requests.
The next useful evidence would be performance over a longer period of routine publishing: article validity, source fidelity, correction rates, time and cost across a wider range of stories. Meyer’s report provides an initial comparison and says the robustness run favored Luna as a practical production option, but it does not give results from a larger independent replication. The account also does not state whether the publisher plans to repeat the test as models or prompts change.
Key Questions
Which model scored highest in the test?
GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 in the 12-article robustness run, according to Meyer.
Why did Meyer choose GPT-6 Luna for production?
Meyer reported that Luna averaged 3.67 in the robustness run, produced valid output for every article in that run and cost about $0.0013 per production article. Astra cost about $0.12 per article, so Meyer judged it too expensive for routine news publishing.
Did higher reasoning effort improve the articles?
Not reliably in Meyer’s test. Scores across the six GPT-6 Luna effort settings were within what he considered noise, while the maximum setting cost and took about 12 times as much as the no-effort setting.
What happened with the backup models?
Meyer said GPT-4.1 nano received low source-fidelity scores and the judges flagged invented quotes. The locally run gpt-oss-120b produced zero valid articles in four attempts, with output-format problems and invented named quotations. Both were removed from the route on October 7.
Can these results establish which model is best for every newsroom?
No. The comparison tested a particular prompt, pipeline, language pair and limited set of sources. Meyer’s results are evidence about that workflow, not a universal ranking across newsrooms or tasks.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
