Business AI, measured on real work.
We gave an AI model 550 real business tasks — sorting an inbox, pulling numbers out of documents, analyzing spreadsheets, writing SQL, drafting replies — and graded every answer against the correct one. Same tasks for every model, three passes each.
What we tested
550 tasks, written to match the work that actually lands on a business desk. Each one has a single correct answer we can grade.
Each task is labeled easy, medium or hard when it is written. One clear step — read the request, produce the answer. Several steps or a judgment call: reconcile records, pick a queue, pull a field. Multi-step work with a trap: a policy to apply, a dataset to analyze, a budget to stay inside.
541 of 550 tasks were answered correctly on all three passes. Two spreadsheet tasks got a different answer on different passes, so the count above is slightly lower than the number of tasks.
Model, quantization, deployment — then the result
A score only means something next to the three things that produced it: which model, how compressed it was, and what it ran on. Change any one of them and it is a different result.
- —Each task is run three times; the score counts all 1,650 attempts, not 550 tasks.
- —541 of 550 tasks were correct on all three passes — two spreadsheet tasks flipped between passes.
- —The misses that remain are real: a policy applied backwards, five hard spreadsheet analyses that ran out of steps, and one answer given as prose instead of the exact value asked for.
Each model we test lands here. Same tasks, same grading, same number of passes.
How fast it answers a single request
With thinking on, the first token arrives at 0.3s, but the visible answer does not start until roughly 24s while the model reasons. The 29s median task time above is that reasoning plus the answer.
One request at a time, no other traffic. A 1,500-word prose memo, capped at 2,000 tokens, streamed three times. Time to first token is measured on the first streamed chunk; words per second is measured over the generation window after that.
Score by kind of work
The tasks it got wrong
Out of 550 graded answers, 542 were correct. These are the misses, described plainly — including the one that is a real judgment error rather than a format slip.
An invoice 42 days old, against a 30-day policy. The model opened a dispute instead of declining. A real miss — it should stay a failure.
Five hard spreadsheet analyses: the model kept re-running the same calculations and used up its work budget. A real limit on long multi-step analysis.
Asked for the supplier code S007, it answered "Supplier S007" in prose. The reasoning was right; the format was not.
Same tasks, same grading, every model
Every model we publish is run against the same 550 tasks with the same correct answers. If the task set changes, every published result is re-run — otherwise the comparison would be meaningless.
Each task is attempted three times. The score counts all the attempts, which is why it reads out of 550 when the task set is 550 tasks.
Quality and speed are different questions. A slow correct answer and a fast wrong one are not the same result, so time is never folded into the score.
Want your model on this table?
We run any open-weight or hosted model against the same tasks and publish the result with its configuration — good, bad, or ugly.
Talk to us