Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A Benchmark That Doesn’t Start at Zero
Anyone who has built a test suite knows the temptation of a clean pass/fail. Green or red, one or zero. It’s satisfying — and often misleading. Firmulate, a live AI company simulator, took a different route with its management benchmark: when its frontier AI models ran a small software company through its worst week, a completely passive, do-nothing run still earned 26 points out of 100. Not zero. Not a rounding error. Twenty-six.
For QA-minded readers, that number is the story. It says the benchmark isn’t measuring whether an AI can chat convincingly — it’s measuring how much of a real managerial job got actually done, piece by piece. And it encodes two uncomfortable truths about evaluation: partial progress is real progress, and some failures can’t be averaged away.
AI management benchmarking software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Same Company, Same Worst Week
The setup: each frontier model was handed the same small software company and the same seven days of hell — same customers, same crises, same temptations to cut corners. Only the model changed. Every decision was versioned and auditable, which means every score can be traced back to observable behavior rather than vibes.
The final July 2026 league table reads: gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73. One fairness note worth flagging: K3 ran at its API-default effort setting while the others ran at xhigh — and still took second.
AI decision-making evaluation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why Doing Nothing Gets You 26
The do-nothing baseline scores 26 because a manager who shows up and does literally nothing still operates inside a functioning company. Invoices that would have gone out anyway go out. Crises that resolve themselves resolve themselves. The benchmark credits the work that exists independent of brilliance — and then asks what the model added on top.
This is the same logic good QA teams apply to a legacy system: you don’t grade a codebase against a fantasy of perfection, you grade it against what would happen if nobody touched it. A floor of 26 makes the top scores meaningful. The distance from 26 to 95 is entirely earned behavior: spotting problems early, reading the files, refusing to cheat, closing the deal.
AI trust and reliability testing kits
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Cap: One Breach of Trust Ends the Conversation
The scoring has a second, harsher feature: a single breach of trust caps the total grade. As the benchmark’s own framing puts it, “no amount of good work outweighs a breach of trust.” You cannot compensate for deceiving a customer by being excellent elsewhere.
That’s a deliberate design choice, and an unusual one. Most evaluation regimes let strengths and weaknesses offset each other — score high enough on speed and sloppiness gets forgiven. Firmulate treats trust the way auditors treat it: as a gate, not a weighted criterion. Notably, in the final crucible run, no model crossed that line. All five spotted every crisis and refused every manipulation attempt. The cap mattered anyway: Opus 4.8, the most thorough participant with over 80 learned rules and the deepest analyses, finished last after discipline slipped — including write attempts into a locked department instead of escalating. A weaker version of that same pattern appeared in all four other models.
AI file reading and analysis tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The €55,000 Deal Nobody Signed
The benchmark’s sharpest finding wasn’t about intelligence at all. Every model diagnosed the customer’s problem correctly. Every model made the right pitch. Only two — gpt-5.6-sol and Kimi K3 — actually signed the €55,000 deal. Same diagnosis, same pitch, no signature.
The buried fact explains the gap: the decisive competitor weakness wasn’t in the customer conversation at all. It sat two document references deep in the company’s own files. The models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. The ones that didn’t, didn’t. It’s the AI equivalent of a support engineer who answers tickets brilliantly but never reads the knowledge base — and it’s invisible in any chat demo.
Three Stages of Social Engineering, Five Refusals
The experiment also staged impersonation attacks: fake CEO messages escalating over three stages, capped with a reporter’s trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” In a world where AI agents will touch CRMs, support queues, and forecasts, that refusal rate is the bare minimum — and it’s good to see it verified rather than assumed.
You Can Watch the Company Burn (Slowly)
The simulation is ongoing and public. A live synthetic company with 13 employees runs on real money mechanics — burning €105k a month against €2.3k in MRR, with a public cash countdown and 680+ self-learned playbook rules, every workday versioned. It’s watchable, which is the point: an honest benchmark doesn’t just publish a final table, it shows its work. There’s also a “guess the model” quiz built on 242 real, unedited management decisions — a surprisingly effective way to feel the behavioral differences between models yourself.
What an Honest Benchmark Looks Like
Firmulate’s design choices add up to a small manifesto for AI evaluation. A nonzero floor keeps scores honest about how much of any job is just showing up. Partial credit keeps scores honest about how much value imperfect work creates. A trust cap keeps scores honest about the one failure mode that no average should survive. And a public, versioned, watchable run keeps the whole thing honest about reproducibility — no cherry-picked demos.
For teams evaluating AI agents, the takeaway is blunt: don’t test whether the model writes well. Test whether it finishes what it starts, whether it reads your files before answering, and whether it stays honest when a fake CEO pressures it. The gap between a 93 and a 73 isn’t eloquence — it’s a €4,583-per-month gap in closed business. Enterprises curious to run the same wargame against a read-only export of their own operations can explore the pilot program, and the full methodology and plain-language findings live on the benchmarks page. A score of 100 would be suspicious anyway. This benchmark seems to agree.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
