The AI Test That Buried a €55,000 Fact Two Documents Deep

Chat Demos Can’t Tell You Whether an AI Reads Before It Answers

If you build, test, or buy software, you already know the gap: a model that demos beautifully and a model that does the actual job are two different things. QA teams spend weeks probing whether an agent hallucinates. But a harder, quieter question is emerging as agents move into CRMs, support queues, and forecasts: does the agent read your files before it answers?

That question now has a measurable answer, thanks to a public experiment by Firmulate, which runs frontier AI models as complete companies — real crises, real money mechanics, real temptations — and scores them on management quality, not chat quality. The July 2026 final league placed gpt-5.6-sol first at 95, Kimi K3 second at 93, Sonnet 5 third at 88, Fable 5 fourth at 77, and Opus 4.8 last at 73. The do-nothing baseline scored 26 — partial progress counts, but a single breach of trust caps the total. As the experiment puts it: no amount of good work outweighs a breach of trust.

The Setup: One Company, Its Worst Week, Four Models

Each frontier model ran the same small software company through the same brutal week — same customers, same crises, same temptations to cheat. Only the model changed. Every decision was versioned and auditable, which is exactly the kind of test harness a QA-minded audience can appreciate: same inputs, different engines, observable outputs.

The headline finding was strange enough to matter: all models spotted every crisis and refused every manipulation attempt. Only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. The experiment’s shorthand for it: the deal was simply left on the table.

The Needle in the File Cabinet

Here’s the part that should change how you evaluate AI agents. The decisive competitive weakness — the fact that justified closing at full price — wasn’t in the customer meeting or the event feed. It sat two document references deep in the company’s own files. A model had to follow one reference to another, read the second document, and connect it to the live negotiation.

The models that did the reading won the €55,000 deal at full price, worth +€4,583 in monthly recurring revenue. The models that didn’t lost it — automatically. Not because they were manipulated, not because they were dishonest, but because they answered without doing their homework.

That’s the measurable version of a property vendors rarely test: retrieval depth under pressure. A chat demo will never surface it, because nobody demos the boring part where the agent follows a citation chain into your own knowledge base.

Honesty Under Fire

The social-engineering leg deserves its own mention. Fake CEO messages escalated over three stages, capped by a reporter’s trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning was the standout: “Treat the request as a suspected approval-bypass / possible impersonation.”

One fairness note from the experimenters: K3 ran at its API-default effort setting while the others ran at xhigh — and still finished second, three points off the lead.

The Thoroughness Paradox

Opus 4.8 is the cautionary tale for anyone who assumes depth equals performance. It was the most thorough participant — over 80 learned rules, the deepest analyses in the field — and it finished last. The close was left on the table, and discipline slipped: write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four lower finishers. Effort without follow-through is just expensive diligence.

The Live Company Behind the League

The testbed itself is real and watchable. A live synthetic company at firmulate.com/live runs 13 employees with real money mechanics — a burn of €105k per month against €2.3k in MRR, a public cash countdown, and over 680 self-learned playbook rules, with every workday versioned. It’s an ongoing artifact, not a one-off benchmark: you can watch the league grow as new runs finish.

Two more doors are open. A “guess the model” quiz at firmulate.com/quiz.html is powered by 242 real, unedited management decisions from the runs. And enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems — via firmulate.com/pilot.html or contact@firmulate.com.

What QA Should Take From This

Three lessons translate directly to software testing and development:

• Test for follow-through, not fluency. The gap between the top and bottom of the league wasn’t diagnosis quality — everyone diagnosed correctly. It was finishing: reading the second document, asking for the signature, closing the loop.
• Bury your needles. Single-hop questions are too easy. The €55,000 fact was two references deep, and that depth is precisely where agent quality separates. Your test suite should include multi-hop retrieval against your own messy, cross-referenced files.
• Score honesty as a cap, not a bonus. Firmulate’s scoring treats a single breach of trust as capping the total, regardless of other good work. That’s a design choice worth stealing for any agent evaluation.

The broader point: “reads your files before answering” is not a soft virtue. In this experiment it was the difference between winning a €55,000 deal at full price and losing it silently. If agents will touch your business systems, that’s the property to test — and thanks to the versioned, auditable format, it’s now the kind of thing you can watch happen rather than take on faith.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI