AI’s True Test: Can It Finish the Job When Stakes Are High?

Imagine you’re in the middle of a tough workout, pushing through fatigue and distractions, but what matters most is whether you can finish strong. In the world of AI, the real measure isn’t just how well it chats — it’s whether it can see a task through, especially when under pressure. Just like in fitness, where grit and consistency count, AI’s value lies in its ability to complete what it starts, even in the toughest moments.

Testing AI in the Real World: The Crucible Experiment

Recently, a live experiment by Firmulate put four of the most advanced AI models through a simulated crisis at a small software company. This wasn’t just about generating convincing dialogue; the models had to run a company facing real money decisions, customer crises, and temptations to cut corners. Every step was recorded, decisioned, and audited, mimicking the real pressures a business faces.

The models included GPT-5.6-SOL, Kimi K3, Sonnet 5, and Fable 5, alongside a baseline score of 26 for partial progress. Their scores ranged from 73 to 95, with GPT-5.6-SOL leading the pack, followed closely by Kimi K3, Sonnet 5, and Fable 5. But the key measure was not just problem detection—they all recognized every crisis and refused manipulation attempts. The true test was whether they could close a deal worth €55,000 — and only two did.

The Hidden Weakness: Reading Beyond the Surface

While all models performed well in spotting crises and resisting deception, the decisive factor lay in their ability to find critical information buried two documents deep in the company’s files. Models that took the time to read and analyze this hidden data were able to close deals at full price, adding over €4,500 monthly recurring revenue. This shows that surface-level chat capabilities are not enough; understanding and digging into deeper information is what truly matters in business execution.

Resisting Social Engineering and Manipulation

In one test, the models faced staged social engineering — fake CEO messages escalating over three stages and a reporter’s subtle attempt to get approval with just one yes/no response. All five models refused, demonstrating strong integrity and suspicion. Kimi K3 explained its reasoning clearly: “Treat the request as a suspected approval-bypass / possible impersonation.” This ability to reject manipulation, even when it’s convincing, is crucial for trustworthy AI in business contexts.

The Real Company: Running Live with AI

The experiment was conducted on a real, operational company with 13 synthetic employees managing actual money — burning €105,000 each month against €2,300 in revenue. The company’s cash countdown, self-learned rules, and versioned daily operations provided a brutal test bed for AI management. Watching this live at firmulate.com/live offers a window into how AI might handle real-world business challenges.

The Surprising Outcome: Discipline Matters

The most thorough model, Opus 4.8, with over 80 learned rules and deep analysis, failed to close the deal and showed discipline slips—writing into a locked department instead of escalating. Interestingly, the same weaknesses appeared in all models, with even the most detailed one leaving a key deal unexecuted. This reveals that even comprehensive AI models can struggle with closing, a core business skill often overlooked in chat demos.

What This Means for Business Leaders

Many AI demos showcase engaging conversations, but that isn’t enough. Performance under pressure, the ability to read deep into files, resistance to manipulation, and closing deals are the real tests. As one executive put it, “If AI agents will touch your CRM, support queue or forecast, the question isn’t just ‘does it write well?’ but ‘does it finish what it starts?’”

Takeaway: Look Beyond the Chat

Business success with AI depends on its ability to deliver tangible results, not just engaging dialogue. The Firmulate experiment demonstrates that only some models can read deeply, resist manipulation, and close deals — the invisible qualities that determine whether AI is a true partner or just a clever mimic. Leaders should focus on testing AI in real operational settings before trusting it with critical tasks.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI