Can The Best AI Model For A News Pipeline Be The Wrong One To Run?

🔍 Read the full analysis: Can The Best AI Model For A News Pipeline Be The Wrong One To Run? on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on real English- and German-language news assignments and chose a cheaper model for production than the one that scored highest. In results published Oct. 13, he reported that GPT-6 Astra led the test, while GPT-6 Luna was selected for routine articles because it cost about one-hundredth as much; two backup models were removed after producing fabricated quotations or invalid output.

The test ran from Sept. 8 to Oct. 13, 2026, using the same production prompt Meyer said his sites use. It covered API, subscription, free-tier and locally run open-weight models. Each assignment drew on one of two real news sources: a 6,340-character product filing or a 619-character political item. Articles were produced in English and German and assessed by two AI judges, GPT-6.1 Sol and Claude Opus 5.5, against the source.

Meyer reports that Astra scored 4.00 out of 5 in both languages in the standard test, with a source-fidelity score of 5.00. Across a further 12-article robustness run, it scored 3.93, with a reported 95% bootstrap interval of 3.83 to 4.02. Luna scored 3.67 in that run, with source fidelity of 4.71 and valid output on every article. Meyer put Astra’s cost at about $0.12 per article, compared with about $0.0012 for Luna; their reported average completion times were roughly 50 and 17 seconds.

The report also says raising Luna’s reasoning-effort setting did not produce a reliable quality gain. At the maximum setting, the reported cost rose to 1.43 cents per article and generation time to 211 seconds, from 0.12 cents and 17 seconds at the “none” setting. Meyer says score differences across effort levels fell within the test’s uncertainty. He also reports that GPT-4.1 nano scored 1.62 in English and 1.68 in German, while the locally run gpt-oss-120b produced zero valid articles out of four.

The Cost of Reliable News Output

The results illustrate why a model leaderboard may not settle a production decision. A publisher running many articles must weigh quality, cost, speed and output reliability, not just the highest test score. In Meyer’s account, Astra’s lead came with a price per article about 100 times Luna’s, while Luna’s reported performance was consistent enough to meet his production needs.

The reported failures among backup models also point to a separate operational risk: a pipeline can produce weak or fabricated copy when its primary model is unavailable and a fallback takes over. Meyer says the judges flagged invented quotations in GPT-4.1 nano’s output and named quotations in gpt-oss-120b’s output. Such results matter because quotation accuracy and source fidelity are core safeguards in news publishing, not merely matters of style.

These findings are from one publisher’s test, not an independent industry-wide comparison. They offer a concrete example of why news organizations may need to evaluate models on their own sources, prompts and publishing constraints rather than assume general benchmark rankings transfer to editorial work.

How the Production Test Worked

Meyer’s evaluation was designed to resemble his own workflow: a source goes in, a finished article comes out under the production prompt. The two source texts varied sharply in length, and Meyer says short sources are typical of his news work. That makes the 619-character item a test of whether a model can write a usable article without filling gaps with unsupported detail.

Two AI judges scored source fidelity, fabrication, clarity about uncertainty, news value, structure and polish; the German edition also received a language-quality score. Meyer says an article counted only if it passed the same output-repair step used in production. He cautioned that, with only two initial test articles, score gaps below about 0.3 points could reflect noise, and said the additional robustness run was the more reliable basis for ranking.

The report also compares performance across different tasks. Meyer says Claude Opus 5.5, which he describes as his leading coding model, scored 3.46 in English and 3.39 in German as a news writer, with source fidelity of 3.75. His account argues that coding performance did not predict news-writing performance in this test. Those scores remain results of his own evaluation, rather than a general determination of which model is best at either task.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his Oct. 13, 2026 report

Limits of the Reported Scores

The results come from a single publisher’s evaluation and two principal source texts, supplemented by a 12-article robustness run for leading configurations. The report does not establish how the models would perform across other publishers’ prompts, subject areas, languages or editorial review processes. Meyer’s use of AI systems as judges also means the scores reflect those judges’ assessments; the report does not describe a human-review comparison.

It is also not clear from the supplied report how the test’s source selection represents the full range of news tasks, or how often each model would fail in long-term production. The reported cost and speed figures are tied to Meyer’s setup and test period and could differ with other usage patterns or pricing. The results should not be read as an independent verification of vendor capabilities or as a general model ranking.

Production Stays With GPT-6 Luna

Meyer says GPT-6 Luna remains the production model at the standard, “none” reasoning-effort setting. He reports that the two cited fallback models were removed from the news route on Oct. 7. The supplied account does not identify a replacement fallback, so it is unclear how the route handles a future Luna outage or failed generation.

The next useful test would be continued monitoring on a broader set of real assignments, including fallback behavior and human editorial checks. Meyer’s report does not announce a new benchmark date or a planned model change. Any future comparison would need to account for updates to model versions, prompts, prices and the mix of sources being processed.

Key Questions

Which model scored highest in Meyer’s test?

Meyer reports that GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test, with source fidelity of 5.00. It scored 3.93 across the 12-article robustness run.

Why did he choose GPT-6 Luna for production?

Meyer says Luna scored 3.67 in the robustness run and produced valid output for every article, while costing about $0.0012 per article compared with about $0.12 for Astra. He judged Luna’s quality sufficient for routine use at a much lower reported cost.

Did more reasoning effort improve Luna’s articles?

Not reliably in the reported test. Meyer says the score differences across six effort settings were within the test’s noise; the maximum setting took about 211 seconds and cost 1.43 cents per article, versus 17 seconds and 0.12 cents at the “none” setting.

What happened with the fallback models?

Meyer reports that GPT-4.1 nano produced invented quotations and received a source-fidelity score of 1.00. The local gpt-oss-120b configuration produced no valid articles in four attempts and also generated named quotations, according to his account. He says both were removed from the news route on Oct. 7.

Does this test prove which AI model is best for news?

No. It reports results from one publisher’s workflow, using a limited set of sources and AI judges. The findings may help explain how to evaluate a particular pipeline, but they do not establish a universal ranking across newsrooms or assignments.

Source: ThorstenMeyerAI.com