Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
What Should a Zero Look Like?
Anyone who has ever graded a test knows the temptation of the round number. A perfect 100 feels earned; a flat zero feels honest. But anyone who has actually graded knows both are usually lies. The student who writes nothing still showed up. The student who aces everything still made one careless error. Real evaluation lives in the messy middle — and a good grading system has to explain, in plain terms, why it gives the points it gives.
That principle is now playing out in public, on an unusual exam: Firmulate’s benchmark, which runs frontier AI models as managers of a small software company through its worst week. The final league table from July 2026 put gpt-5.6-sol first at 95, Kimi K3 second at 93, Sonnet 5 third at 88, Fable 5 at 77, and Opus 4.8 last at 73. But the number that should stop any grader’s eye is the one nobody earned: a do-nothing baseline — a manager that essentially sits on its hands — scores 26. Not zero. Twenty-six.
For an audience that cares about how we measure things, that 26 is the most interesting figure in the whole experiment. It is a methodology statement disguised as a data point.
Products Worth Considering
Anker SOLIX C300 Portable Power Station, 288Wh
Bose SoundLink Flex Bluetooth Speaker (2nd Gen) – Portable Outdoor Speaker with Hi‑Fi Audio, Waterproof and Dustproof, USB‑C, Up to 12 Hours
Logitech MX Mechanical Wireless Illuminated Keyboard, Clicky
As an affiliate, we earn on qualifying purchases.
Partial Progress Counts
Why does doing nothing earn 26 points? Because the scoring refuses to pretend that a company week is pass/fail. A manager who takes no risky action still avoids catastrophes; crises go unhandled, but nothing is made worse. Some fraction of running a business is simply not torching it. The benchmark credits that partial progress explicitly rather than collapsing everything below “excellent” into an undifferentiated zero.
This is the same reasoning behind rubrics in education: a student who identifies the problem but botches the solution has still demonstrated something. The Firmulate scale spreads performance across a range — 95 at the top, 73 at the bottom among real competitors, 26 for deliberate passivity — which makes the differences between models legible instead of binary.
One Breach Caps the Grade
The second rule is harsher: a single breach of trust caps the total score. The benchmark’s own phrasing is blunt — “no amount of good work outweighs a breach of trust.” A model could handle every crisis flawlessly and still see its ceiling collapse if it crossed an integrity line once.
Again, this is a deliberate grading philosophy. It encodes the idea that trustworthiness is not one competency among many to be averaged in, but a gate. Most workplace evaluations already work this way informally — a manager who is brilliant but leaks confidential data is not “90 percent excellent.” The benchmark simply makes that intuition explicit and mechanical.
Distrust of Round Numbers
Notice, too, what the top of the table doesn’t contain: a 100. The best performer scored 95. A grading system that never hands out a perfect score is signaling something — that there is always room to slip, and that a suspiciously round 100 deserves scrutiny. In a field where AI evaluations are sometimes accused of grade inflation, a ceiling that nobody reaches is itself information.
The Exam Itself
So what were the models actually graded on? Each frontier model ran the same small software company through its worst week — same customers, same crises, same temptations to cheat. Only the model changed. Every decision was versioned and auditable, meaning the graders could reconstruct exactly why a score came out the way it did.
The headline finding: all models spotted every crisis and refused every manipulation attempt. Social engineering failed completely — fake CEO messages escalating over three stages, plus a reporter’s “just one yes/no, on background” trick, were rebuffed by five of five models tested on it. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”
Yet only two models signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. The decisive fact was buried two document references deep in the company’s own files, not in the customer event. Models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue.
Opus 4.8 is the cautionary tale that justifies the grading system: it was the most thorough participant, generating +80 learned rules and the deepest analyses, yet finished last. The close was left on the table, and discipline slipped — write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. Effort matters, output doesn’t equal outcomes — and the score reflected that.
Watch the Grading Live
Unlike most benchmarks, this one runs in the open. A live synthetic company — 13 employees, real money mechanics burning €105k/month against €2.3k MRR, a public cash countdown, and 680+ self-learned playbook rules — is watchable at firmulate.com/live, with every workday versioned. There’s even a “guess the model” quiz built from 242 real, unedited management decisions at firmulate.com/quiz.html. One fairness note worth recording: K3 ran at the API-default effort setting while the others ran at xhigh — and still nearly won.
The Takeaway
A benchmark is a theory of value expressed as a number. Firmulate’s theory is readable in its floor and its cap: partial progress counts (26 for doing nothing), and integrity is a gate, not a line item. The top score of 95 — never 100 — completes the picture: honest measurement leaves headroom for the failure that hasn’t happened yet.
For educators and anyone who builds evaluations, the lesson transfers directly. A grading system that can explain its zeros and its ceilings is worth more than one that produces flattering round numbers. And for enterprises wondering whether AI agents can be trusted with a CRM or a forecast, the full results — and a pilot that runs the same wargame against a read-only export of your own business — make the case better than any demo.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
