Imagine training an AI for your woodworking shop, only to find it can’t finish the job or even stay honest when under pressure. That’s what’s happening in the latest AI benchmark, where even a do-nothing baseline scores 26 points out of 100. For DIYers and small business owners alike, understanding how AI handles real-world crises — not just chat fancy — is crucial. Because when AI touches your tools, your workflow, or your customer trust, it’s about more than just what it says; it’s about what it actually does.
Get business pricing on tools and workshop supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Understanding the Benchmark: More Than Just Chatting
In the latest Firmulate experiment, four advanced AI models were tasked with managing a small, simulated software company during its worst week — the same customer crises, same temptations, same challenges. The goal was simple but revealing: Can these models handle real decisions, stay honest, and deliver real value? The results were eye-opening.
The Baseline Score: Why 26 Points?
One surprising figure emerged: a do-nothing baseline, which essentially makes no decisions, scored 26 points. That’s because the scoring system counts even partial progress. It recognizes that doing nothing is better than making mistakes or attempting to manipulate the system. Moreover, a single breach of trust — like signing a deal without reading the details — caps the total score at that level, confirming that honesty and diligence matter more than just good intentions.
Honest AI Wins in the End
All four models managed to spot every crisis and refused manipulation attempts, including social engineering tricks like fake CEO messages. Interestingly, only two of them closed a deal on the company’s own analysis — and only after reading critical internal documents. The twist? Those documents were hidden two layers deep in the company’s files; reading them was the key to sealing the deal at full price, worth over €4.5k in monthly recurring revenue.
Why Reading Deep Matters
Most models performed well at surface-level tasks but faltered where deeper understanding was needed. The most thorough participant, Opus 4.8, with over 80 learned rules, still left the close on the table due to slipping discipline — like attempting to escalate issues to a locked department rather than following proper procedures. This weakness was common across all models, highlighting that thoroughness and adherence to protocols are vital for real-world AI performance.
Trust and Discipline Over Raw Power
The experiment also tested social engineering resilience. All models refused staged fake CEO messages, treating them as impersonation risks. Meanwhile, the models’ fairness settings varied: K3 ran without an effort parameter, while the others operated at high levels of responsiveness, which could influence decision quality. Yet, even with these differences, the core takeaway was clear: honesty, discipline, and proper process adherence are the true measures of effective AI management.
Popular right now
Ninja Foodi 6-in-1 DualZone Air Fryer, 8 qt, Grey (DZ201)
Philips 2200 Series Fully Automatic Espresso Machine with Classic Milk Frother, 2 Coffee Varieties, Intuitive Touch Display, Ceramic Grinder
Sony BRAVIA XR8B 65-Inch 4K OLED Google TV (K-65XR8B)
As an affiliate, we earn on qualifying purchases.
Implications for Your Business
For small and DIY businesses, the message is straightforward: AI isn’t just about generating content or answering questions — it must finish what it starts, read the relevant files, and stay honest when under pressure. The current leaderboard in this live, watchable experiment shows that even the best models are not perfect; they refuse manipulation and can miss details that are critical for closing deals or making decisions worth thousands of euros every month.
Furthermore, the experiment underscores that partial progress counts — a model that refuses to sign a bad deal or cut corners scores higher than one that might close a deal but breaches trust. This reflects a broader principle: in real-world operations, integrity and attention to detail are paramount.
Watch the Live Experiment
The live experiment is ongoing at firmulate.com/live. You can observe these AI models managing a synthetic company, making decisions, and navigating crises in real time. This transparency allows business owners and DIY enthusiasts to see exactly how AI performs beyond the chatter, especially when stakes are real and money is on the line.
Final Thought: Not All AI Is Created Equal
The lessons from this benchmark are clear: AI models that handle your business interactions must do more than just respond well — they need to finish tasks, read critical information, and stay honest under pressure. The scoring system’s minimum of 26 points for doing nothing serves as a reminder that baseline performance isn’t zero — it’s a starting point for genuine trustworthiness. For DIYers and small business owners, that means choosing AI tools that are tested in realistic scenarios, not just in demos. Because when it comes to managing your work, honesty and diligence are the best tools you can have.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
