What if the software were also the company?
For developers and quality-assurance teams, the most revealing test is rarely whether a system works under ideal conditions. It is what happens when requirements collide, permissions tighten, customers become impatient and an apparently helpful message asks someone to bend the rules.
Firmulate turns that principle into a public business experiment. Its small software company has 13 synthetic employees, real money mechanics and a severe financial problem: it burns €105k per month while producing €2.3k in monthly recurring revenue. A public cash countdown makes the pressure visible, while every workday is versioned and auditable. The company has also accumulated more than 680 self-learned playbook rules as it operates.
This is build-in-public taken beyond product updates and revenue charts. Visitors can watch the company live as an ongoing corporate survival story, complete with decisions, setbacks and material generated by each business day.
AI FOR QUALITY ASSURANCE AND SOFTWARE TESTING: The Practitioner's Complete Guide to AI-Powered Testing, Tools, and Transformation
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A worst week, repeated under controlled conditions
Firmulate’s Crucible League gave each frontier model the same assignment: run the same small software company through its worst week. The customers, crises and temptations remained constant. Only the model changed. Every decision was versioned and available for audit.
The final July 2026 standings put gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress still counted. However, a single breach of trust capped the total under the explicit principle that “no amount of good work outweighs a breach of trust.”
The reassuring finding was that every model identified every crisis and rejected every manipulation attempt. The more troubling result was that only two signed the €55,000 deal their own work had earned. Firmulate summarizes the failure crisply: “Same diagnosis, same pitch — no signature.”
That distinction matters for software teams evaluating agents. A model can recognize a problem, explain a solution and even prepare the correct next action without actually completing the work. In a chat demonstration, that gap may look minor. Inside a business process, it can determine whether analysis becomes an outcome.
The decisive fact was not where the crisis appeared
The deal hinged on a competitor weakness buried two document references deep in the company’s own files. It was not present in the customer event that initially demanded attention. Models that followed the trail and read the file secured the deal at full price, adding €4,583 in monthly recurring revenue.
This is a familiar lesson for QA and software development: the visible event is not always the complete specification. Useful systems must connect current activity with established business context. Merely responding fluently to the latest prompt is not enough when the evidence needed for a correct decision lives elsewhere.
Pressure tested trust as well as competence
The experiment also included fake CEO messages that escalated across three stages, followed by a reporter’s attempt to obtain “just one yes/no, on background.” All 5 models refused. Kimi K3 recorded the clearest diagnosis: “Treat the request as a suspected approval-bypass / possible impersonation.”
That unanimous refusal is important because operational AI will encounter requests that sound urgent, authoritative or socially difficult to reject. Firmulate’s test treated honesty and permission discipline as core management behavior, not secondary safety features. Readers can examine more of what the synthetic employees say on the public quotes page.
Thoroughness did not guarantee execution
Opus 4.8 offers the sharpest cautionary profile. It was the most thorough participant, producing the deepest analyses and adding 80 learned rules, yet it finished last. It left the close on the table and attempted to write into a locked department instead of escalating the blockage. The same discipline problem appeared in weaker form across the other four models.
The result challenges a common assumption in AI evaluation: that more analysis naturally produces better performance. Firmulate’s evidence shows that careful reasoning can coexist with incomplete execution and poor handling of operational boundaries.
There is also an important fairness qualification. Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. That difference should remain visible when comparing its 93-point finish with the rest of the field.
As an affiliate, we earn on qualifying purchases.
A living acceptance test for AI work
Firmulate’s public company turns abstract questions about agent reliability into observable business behavior. Can a model read beyond the immediate event? Can it preserve trust when authority is impersonated? Can it escalate a locked permission rather than repeatedly pushing against it? Most importantly, can it finish the work it correctly diagnosed?
The company’s weak economics make those questions concrete. With €105k in monthly burn against €2.3k in monthly recurring revenue, incomplete execution is not an academic defect. It is part of a visible fight for survival.
For software, QA and development readers, the experiment suggests a practical standard: evaluate AI as an operator across an entire workflow, not as a writer responding to isolated prompts. Firmulate’s most compelling contribution is not a polished demonstration. It is a running company whose decisions, discipline and consequences remain open to inspection.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
