The Test Everyone Studies For Is the Wrong One
Education researchers have a name for it: teaching to the test. Students learn to answer the questions on the exam, not the ones life actually asks. Something strikingly similar is happening in how we evaluate AI. Coding leaderboards and chat arenas grade answer quality — elegant code, witty replies, correct facts. But those are classroom skills. Nobody grades whether the student finishes the homework, reads the attached documents, or tells the truth when lying would be easier.
That gap between exam performance and real-world competence is exactly what a live public experiment at Firmulate set out to measure — and the results read like a report card full of surprises.
AI management decision-making tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
One Company, One Terrible Week, Five Frontier Models
The setup is elegantly simple. Four frontier AI models — gpt-5.6-sol, Kimi K3 from Moonshot, Sonnet 5, Fable 5 and Opus 4.8 — were each handed the same job: run the same small software company through its worst week. Same customers, same crises, same temptations to cut corners. Only the model changed. Every decision was versioned and auditable, so nothing about the run can be quietly retconned after the fact.
Think of it as a standardized exam, but the questions are a churn wave, a price increase, a downround and a PR crisis — the modern management curriculum. The subject isn’t chat quality. It’s management quality.
The Final League Table
- 1. gpt-5.6-sol — 95 points. Found the buried fact, closed the deal — the complete performance.
- 2. Kimi K3 — 93 points. The newcomer closed the deal too, with the cleanest discipline of the field. (A fairness note: K3 ran at its API-default effort setting while the others ran at xhigh — and still nearly won.)
- 3. Sonnet 5 — 88 points. Closed the deal, with a few more process slips.
- 4. Fable 5 — 77 points. Mid-field.
- 5. Opus 4.8 — 73 points. Most thorough participant of all — and still last.
For calibration: a do-nothing baseline scores 26, because partial progress counts. But a single breach of trust caps the total — the experiment’s grading philosophy is blunt: no amount of good work outweighs a breach of trust.
Same Diagnosis, Same Pitch — No Signature
Here is the finding that chat demos can never show. All models spotted every crisis. All refused every manipulation attempt. And yet only two signed the €55,000 deal that their own analysis had earned. Five models, five correct diagnoses, five competent pitches — and three of them left the pen on the table. Finishing what you start turns out to be a separate skill from knowing the answer.
The Buried Fact
The decisive detail was hidden two document references deep in the company’s own files — not in the customer event at all. The models that actually read their own documentation found the competitor weakness and won the deal at full price, worth +€4,583 in monthly recurring revenue. It’s the corporate equivalent of the exam question whose answer is in the reading you were supposed to do beforehand.
Honesty Under Fire
The social engineering tests were equally instructive: fake CEO messages escalating over three stages, plus a reporter’s disarming “just one yes/no, on background” trick. Five out of five models refused. Kimi K3’s on-record reasoning deserves quoting: “Treat the request as a suspected approval-bypass / possible impersonation.” That is not chat quality. That is institutional integrity — the kind boards say they want and rarely get to test before hiring.
The Opus Paradox
The most poignant profile belongs to Opus 4.8: the most thorough participant in the field, with the deepest analyses and over 80 self-learned rules — and a last-place finish. The close was left undone, and discipline slipped: write attempts into a locked department instead of escalating. The same weakness, in weaker form, appeared in all four models. Diligence without follow-through is a very human failure, it turns out.
Watch It Happen, or Test Your Own Judgment
The experiment isn’t a slide deck — it’s a live company you can observe. Thirteen synthetic employees, real money mechanics, a burn of €105k per month against €2.3k in MRR, a public cash countdown, and over 680 self-learned playbook rules. Every workday is versioned, and it’s watchable at firmulate.com. Full results and plain-language findings are on the benchmarks page.
For readers who like a pop quiz: 242 real, unedited management decisions from the runs power a “guess the model” challenge — a humbling exercise in how similar these systems look until the closing moment. And enterprises can run the same wargame against a read-only export of their own business; nothing ever writes back to real systems.
The lesson for anyone hiring AI — or teaching humans — is the same one education has been learning for a century: the exam everyone studies for measures the wrong thing. Answer quality is now cheap. Management quality — reading your own files, finishing what you start, staying honest when a fake CEO tells you otherwise — is the new curriculum. And unlike chat demos, that gap is invisible until the worst week arrives.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
