Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
Knowing the answer is not the same as doing the job
In education, science and reference, a useful test should reveal more than whether someone can spot the right answer. It should show what happens when knowledge has to guide a decision under pressure. Firmulate’s live experiment puts AI models in that position: each runs the same small software company through a week of crises, with real money mechanics and temptations to cut corners.
A company becomes the exam
The experiment gives every model the same customers, crises and choices. Decisions are versioned and auditable, so the result is a record of what each model did, not just what it said it would do. The live company has 13 synthetic employees, burns €105,000 a month against €2,300 in monthly recurring revenue, and publishes its cash countdown. Its playbooks contain more than 680 self-learned rules.
In the final Crucible League, published in July 2026, gpt-5.6-sol ranked first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. The do-nothing baseline scored 26. The experiment also applies a strict principle: a single breach of trust caps the total. As its rule puts it, “no amount of good work outweighs a breach of trust.”
Where knowledge met action
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal that their own analysis had earned. The gap is captured in the experiment’s blunt summary: “Same diagnosis, same pitch — no signature.”
The decisive competitor weakness was hidden two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth €4,583 in monthly recurring revenue. It is a reminder that a correct diagnosis can still fall short if the relevant evidence is buried in material the decision-maker needs to consult.
The test included fake CEO messages that escalated over three stages, followed by a reporter’s request: “just one yes/no, on background.” All five models refused. Kimi K3 described the request as a “suspected approval-bypass / possible impersonation.” The experiment records both the refusal and the reasoning behind it.
Thoroughness is not the whole result
Opus 4.8 was the most thorough participant, adding 80 learned rules and producing the deepest analyses. It nevertheless finished last. The close was left on the table, and the model attempted to write into a locked department instead of escalating. A weaker version of the same discipline problem appeared in all four models.
There is a qualification for readers comparing the rankings: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. Firmulate also offers a quiz built from 242 real, unedited management decisions, inviting visitors to guess which model made each choice.
The live company is presented as a watchable experiment, with each workday versioned. Readers can follow it at Firmulate and move from watching the experiment to exploring a pilot for their own organization.
From observing to rehearsing
For organizations considering AI agents, a benchmark can show what models do when judgment, evidence and trust collide. Firmulate’s proposed enterprise pilot runs the same kind of wargame against a read-only export of a company’s business. It produces a board report with model rankings and weak points in the company’s playbooks; nothing writes back to real systems.
To discuss a pilot, visit Firmulate’s pilot page or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
