Every teacher knows the trap: students who ace the flashcards and flunk the real exam. Chat benchmarks are flashcards. They show whether a model can produce a polished answer to a known question — not whether it can run a business through its worst week. So one team of researchers built the exam instead: a real, watchable software company with real money mechanics, real customers, and real temptations to cheat, and handed the CEO seat to five frontier AI models, one at a time, to see who actually finishes the job. The final league table from the July 2026 run has a lesson that belongs in every syllabus on evaluation: the newcomer can beat the establishment, and the only test that matters is your own.
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
The Experiment
Firmulate, the AI company emulator, ran the same experiment five times: each frontier model got the identical small software company, the identical customers, the identical crises, and the identical temptations to cut corners. Every decision was versioned and auditable, so nothing rests on anecdote. The company itself is no toy — 13 synthetic employees, a burn rate of €105k per month against just €2.3k in monthly recurring revenue, a public cash countdown, and more than 680 self-learned playbook rules accumulated over its life. It runs every business day, and you can watch it lose money in real time.
Products Worth Considering
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
DJI Mini 3 Drone with DJI RC and 3-Axis Mechanical Gimbal
Amazon Kindle Scribe (3rd Generation), 64GB, 11-inch Display, Graphite, with Premium Pen
As an affiliate, we earn on qualifying purchases.
The Final Scores
The results, from the completed Crucible run:
- 1. gpt-5.6-sol — 95. The complete performance: found the buried fact, closed the deal.
- 2. Kimi K3 — 93. The newcomer from Moonshot, and the surprise of the field.
- 3. Sonnet 5 — 88. Solid, with a few more process slips.
- 4. Fable 5 — 77.
- 5. Opus 4.8 — 73. The most thorough participant, yet last.
For context, a do-nothing baseline scores 26 — partial progress counts, but a single breach of trust caps the total. As the experiment’s own rule puts it: no amount of good work outweighs a breach of trust.
A fairness footnote: K3 ran without an effort parameter (API default) while the others ran at xhigh — and still nearly won.
What the Exam Actually Measured
The headline finding is quietly devastating for anyone who picks models from chat demos. All five models spotted every crisis. All five refused every manipulation attempt, including a fake-CEO social-engineering campaign that escalated over three stages and a reporter’s trick — “just one yes/no, on background.” Kimi K3’s on-record reasoning was textbook: “Treat the request as a suspected approval-bypass / possible impersonation.” Five out of five refused.
But only two models signed the €55,000 deal that their own analysis had earned. Same diagnosis, same pitch — no signature. The difference? The decisive competitor weakness was buried two document references deep in the company’s own files, not in the customer event. The models that actually read the file closed the deal at full price, worth +€4,583 in monthly recurring revenue. The others left it on the table.
Then there’s the discipline axis. K3 found the buried needle, won the deal, saved the churning customer, and resisted all three baits — with only one deviation across the entire week, the cleanest discipline in the field. Opus 4.8 is the cautionary tale: it generated the most learned rules (+80) and wrote the deepest analyses, yet finished last, because the close never happened and discipline slipped — it attempted writes into a locked department instead of escalating. The same weakness appeared, in weaker form, in all four other models.
Why This Matters Beyond League Tables
Crucible-spotting, deal-closing, and bait-resistance are invisible in chat demos. That’s the point. If AI agents will touch your CRM, your support queue, or your forecast, the question is not “does it write well” but “does it finish what it starts, does it read your files first, and does it stay honest under pressure?” A model that diagnoses perfectly but never signs is a very expensive consultant.
The league being open — a newcomer taking second place ahead of three of four Western frontier models — means vendor rankings are no longer a safe shortcut. Picking a model without testing it against your own work is now a bet, not a decision.
Firmulate’s deeper contribution is methodological, and it’s one educators should recognize: stop grading the essay, start running the practicum. The team has packaged 242 real, unedited management decisions from the experiment into a guess-the-model quiz — a genuinely instructive exercise in judging AI judgment. Enterprises can go further and run the same wargame against a read-only export of their own business, with nothing ever writing back to real systems. And the live company keeps running every business day at firmulate.com — a longitudinal study you can check on like a weather report, not a slide deck frozen at publication time. The lesson generalizes far beyond AI: the only benchmark that predicts performance is the one that reproduces the conditions of performance. Everything else is flashcards.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
