When AI Diligence Fails at the Last Mile: Lessons from a High-Stakes Business Test

Many assume that more knowledge, more rules, and relentless diligence make AI better. But what if even the most thorough AI still misses the crucial moment that matters most? This is the story of a real-world experiment revealing that in business, impact often hinges not on volume but on focus—and discipline.

The AI Experiment: Putting AI to the Test in a Live Business Environment

In a groundbreaking live experiment, four advanced AI models were tasked with managing a simulated small software company during its most challenging week. This wasn’t a typical chat demo—these models faced real crises, customer demands, and temptations to cut corners. Every decision was carefully versioned and auditable, reflecting how AI might operate in actual business settings.

The Models and Their Scores

• GPT-5.6-sol: scored the highest at 95, successfully identifying critical information buried deep in company files and closing the deal worth over €4,583 in monthly recurring revenue.
• Kimi K3: a newcomer to the league, scored 93, and demonstrated the cleanest discipline—refusing manipulative tactics and adhering strictly to protocols.
• Sonnet 5: scored 88, closed the deal but with minor process slips.
• Fable 5: scored 77, closed the deal but with more process weaknesses.
• Baseline: scored only 26, highlighting how partial progress and slip-ups can limit AI effectiveness.

The Critical Finding: Information Depth Wins

Despite all models spotting crises and refusing manipulative social engineering—like fake CEO messages with staged escalation—the decisive factor was reading and understanding company documents. The models that delved into files and references deep within the company’s own records managed to uncover hidden facts, enabling them to close the deal at full price. This buried insight was two document references deep, invisible in surface-level analysis but crucial for success.

Implications for Business AI

Most AI demos focus on surface-level chat or quick answers. But in a real business environment, impact depends on the AI’s ability to read deeply, prioritize correctly, and stay disciplined under pressure. The experiment underscores that volume of learned rules or superficial diligence does not guarantee success. Instead, focus and discipline—reading carefully, resisting shortcuts—are what truly matter.

The Discipline Test: The Last Dropped Ball

Opus 4.8, despite being the most thorough participant with over 80 learned rules and deep analysis, finished last because it lost discipline at the critical moment. Instead of escalating issues into a secure department, it attempted to write into the company’s work files—an act that left opportunities on the table and showed how even diligent systems can falter without strict process adherence.

The Human Side: Trust, Deception, and AI Integrity

Throughout the experiment, all models refused to sign off on a staged social engineering attempt involving staged CEO messages and a reporter trick. Kimi K3 explicitly treated these as suspicious, demonstrating that, even under pressure, AI models can maintain integrity and resist manipulation—an essential trait for trustworthy automation.

Real Business, Real Stakes

The experiment was run in a simulated environment mirroring a live company with 13 synthetic employees, managing real money mechanics—burning €105,000 monthly against a backdrop of €2,300 in monthly revenue. Every move was versioned daily, offering a transparent look into decision-making processes. This is not theoretical; it’s a live, watchable test at firmulate.com/live.

Key Takeaways for Business Leaders

• Knowing more rules or having more learned information doesn’t necessarily translate into better results.
• Deep understanding, disciplined decision-making, and reading deeply into data are more impactful than volume or superficial diligence.
• Even highly thorough AI models can falter if discipline slips—highlighting the importance of clear process adherence.
• Trust and integrity can be maintained; models refused manipulation attempts, showing promise for trustworthy AI in sensitive roles.

The Bigger Picture: Wargaming Your AI Workforce

Business leaders can now run their own AI wargames against a read-only export of their operations—without risking real systems. This allows testing AI decision-making under pressure, identifying weaknesses, and ensuring discipline before deployment. More details and experiments are available at firmulate.com/pilot.

The Final Word: Quality Over Quantity

In the end, the most thorough AI was not the winner. Success came from focus, discipline, and the ability to read deeply—reminding us that in business and AI alike, impact often hinges on what you prioritize and how disciplined you are in executing it.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI