What AI Can Really Do for Business — Beyond Chat
In the rapidly evolving landscape of artificial intelligence, it’s tempting to judge AI models by their ability to produce convincing conversations or generate content. But in the real world — especially in high-stakes business decisions — the true test is whether AI can deliver results under pressure, stay honest amidst temptations, and execute on its own analysis. A groundbreaking live experiment by Firmulate puts this question front and center, revealing surprising insights about AI’s practical capabilities in business management.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Live Business War Game: Putting AI to the Test
In a unique, real-world experiment, four advanced AI models were tasked with running a small software company through its worst week. This simulated environment included the same customers, crises, and temptations — such as manipulation attempts and social engineering — for each model. The goal? To see which AI could diagnose problems, resist manipulation, and actually close a €55,000 deal, based solely on its own analysis and decision-making.
The models—gpt-5.6-sol, Kimi K3, Sonnet 5, and Fable 5—were evaluated on multiple fronts: their crisis detection, discipline, honesty, and ability to follow through. Every decision was versioned and auditable, ensuring transparency in their actions. The results revealed a stark contrast between what AI can say and what it can actually do in practice.
Findings: All AI Models Spot Crises and Reject Manipulation
Remarkably, all four models identified every crisis and refused every attempt at manipulation, including social engineering tactics like fake CEO messages and reporter tricks. For instance, when asked to approve a questionable action through a staged impersonation, all models refused, with Kimi K3 explaining: ‘Treat the request as a suspected approval-bypass / possible impersonation.’ This demonstrates that current AI models are capable of robust defensive behavior when it comes to manipulation tactics.
The Hidden Weakness: Execution and Discipline
However, the story shifts dramatically when it comes to execution. Only two models—gpt-5.6-sol and Kimi K3—actually closed the deal, signing the €55,000 contract earned through their own analysis. The other two models, despite diagnosing the issues correctly, failed to follow through. One, Fable 5, left the closing on the table and shifted discipline into a locked department instead of escalating it properly.
Digging deeper, the decisive factor was the models’ ability to read and interpret the company’s own files. The models that read two document references deep into the company’s files—finding buried facts—won the deal at a full €4,583 MRR (monthly recurring revenue). This buried information was the key to success, yet it remained invisible in traditional chat demos that focus solely on surface-level interactions.
Why This Matters: Chat Isn’t the Whole Story
The experiment underscores a crucial point for anyone deploying AI in real business settings: measuring chat quality is not enough. The true measure of an AI’s business utility is whether it can actually finish what it starts, read relevant internal documents, and stay honest under pressure. These capabilities are invisible in casual chat demos but are essential for effective management and decision-making.
The Broader Implications: Trust, Execution, and Cost
In the live experiment, the models’ scores ranged from a high of 95 for gpt-5.6-sol to 77 for Fable 5. The top scorer found and closed the deal by uncovering a buried fact in the company’s files, demonstrating that deep comprehension and execution discipline are vital. Meanwhile, models running without effort parameters (like Kimi K3 at default settings) performed almost as well, hinting at how user configuration impacts trust and performance.
Ultimately, the key takeaway is that AI’s value is not just in generating convincing language but in reliably executing complex, high-stakes tasks. The experiment also highlights the importance of testing AI models in scenarios that mimic real-world pressures and temptations—something chat demos cannot reveal.
See It in Action and Run Your Own Tests
The live company’s operations are transparent — from its daily decision-making to its cash position of burn €105k/month against just €2.3k MRR. The experiment is watchable at firmulate.com/live, where you can see the decisions unfold in real time. You can also explore the benchmarks to understand how different models perform and run your own ‘wargame’ against your business using a read-only export. This approach enables companies to evaluate their AI workforce before fully integrating it, ensuring reliability and integrity.
Key Takeaway
While AI chat demos impress with surface-level dialogue, the real test of AI in business is its ability to read internal documents, resist manipulation, and follow through on commitments. A live experiment shows only some models can do this reliably, emphasizing that true AI utility lies in execution, discipline, and trustworthiness—qualities that are invisible without real-world testing.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
