What we test, and how it is graded.
Every model runs the same 550 tasks, three times each, against answers we can check mechanically. This page is the whole method — read it once and every score on the leaderboard means the same thing.
Ranked on correctness first. A model that gets every answer right but takes ten minutes beats one that is quick and wrong — the wrong answer is still wrong when it lands. Speed is published next to every score so you can see the trade-off, but it never moves a model up the ranking. A fast wrong answer is a cost, not a result.
What we test
550 tasks, written to match the work that actually lands on a business desk. Each one has a single correct answer we can grade — the test is not whether a document got produced, it is whether the answer is right.
Each task is labeled easy, medium or hard when it is written. One clear step — read the request, produce the answer. Several steps or a judgment call: reconcile records, pick a queue, pull a field. Multi-step work with a trap: a policy to apply, a dataset to analyze, a budget to stay inside.
How we grade it
Every task has a key, and the key is checked mechanically — no model judging another model's prose. A generated document is worth nothing if it says the wrong thing, so the grading asks what decision was made, whether it cites the right rule, and whether it promises anything the policy forbids.
The queue it should go to, and the outcome the policy gives it.
Both answers compared against the key. Correct on both counts scores 1. Right on one of the two scores 0.5 — it is graded as wrong overall, but the partial credit is recorded.
Every field on an invoice, purchase order or contract — numbers, dates, SKUs, line items, totals.
Field by field against the generated document. Dates must match as dates, money as cents, and line items as rows — so 1,216.90 in the wrong place does not pass.
A single number, count, percentage, list or set computed from the company's sales ledger.
Compared to the answer computed from the same files. Money is allowed 0.5% relative tolerance and one cent absolute; counts must match exactly; percentages must be within 0.1 of a point.
A query or a spreadsheet formula.
The query is actually run against the company database, and its rows are compared to the reference query's rows. Row order matters only when the question asked for an order. A query that errors scores zero.
A draft customer email, plus the outcome, the policy section and the reference it cites.
Four separate checks, all of which must pass: the outcome the policy gives, the correct policy section, the reference number appearing inside the email body the customer would read, and no promise the policy forbids for that case (for example offering a refund when a restocking fee applies). Prose quality is not scored by a model — the draft either states the right decision with the right citation and no forbidden promise, or it does not.
The account rep a request belongs to.
Exact match against the key.
Each task is run three times and all three attempts are graded. 550 tasks × 3 passes is 1,650 graded answers — which is why the suite-wide score reads out of 1,650, while each model page reads out of 550.
How to read a result
Graded answers correct. Not tasks completed, not documents produced — answers that match the key.
Every result is printed with the model, the quantization, the deployment and the source it came from. Change any one and it is a different result, so a score never stands alone.
Time to first token, words per second, and how long a typical task takes. Quality and speed answer different questions and are never folded together.
Four real documents produced once per model, graded on the files themselves and published on that model's page. The quality score is an addendum judged against a published rubric — it never changes whether a file passed.
Speed figures are measured per model — single request, nothing else running — and printed on that model's page, never averaged across models.
The files round
The finale is the one part of this benchmark where the model produces real files rather than an answer. One fixed fact sheet, one brand kit, four artifacts — the way a person would actually be asked to do it. It runs once, because nobody regenerates an annual report a hundred times.
Each model's files are published, unedited, on its own page — open a model from the leaderboard to download them.
Acme Co's FY2025 results against FY2024 — revenue, margin, twelve months of trading, ten product categories, the top twenty customers and the geographic split. Every figure in every artifact reconciles to the same fact sheet.
An 8–12 page annual report with six required sections: shareholder letter, financial review, category performance, customer and geographic performance, risk factors, and outlook.
- —Every headline figure reconciles
- —All six sections present
- —Logo on the document
- —Headings in the brand colour
A two-sheet workbook: a Summary comparing FY2025 to FY2024, and a Monthly sheet with all twelve months.
- —Totals are real formulas, not typed numbers
- —Change and growth are formulas that reference cells
- —All twelve months present
- —Figures match the fact sheet
A ten-slide investor presentation following a fixed outline, with the brand palette applied and at least one real chart object.
- —Ten slides in the required order
- —Figures correct on every slide
- —A chart object, not a picture of one
- —Logo on every slide
- —Palette applied
An X post of 280 characters or fewer and a LinkedIn post, both announcing the results, text only.
- —Within both character limits
- —Headline figures present
- —No guidance, forecast or claim the fact sheet does not support
Pass or fail is machine-checked against the fact sheet and the brief — no model decides whether a document passes. Separately, a quality score comes from Claude Opus 5.5 against a published four-part rubric, run twice on each artifact. It is reported as an addendum and never folded into the pass rate.
Does this read as a designed document with clear hierarchy?
Is the brand applied with restraint and competence — colour, charts, logo?
Does the prose tell a coherent story rather than list numbers?
Could this be handed to an investor or board without embarrassment?
- —All three models produce properly formatted workbooks — bold headers, fills, borders, currency formats, sized columns and live formulas. Nemotron edges the group (2.75) for consistent percent formatting; Qwen loses ground (2.12) on inconsistent percent formatting. None of them writes interpretive prose into the sheet, which is why narrative quality stays low across the board.
- —Qwen's deck is the strongest single artifact in the set (3.62 — the only one above 3 on all four rubric axes). It built it with reasoning on, the setting that produced no deck at all for Nemotron.
- —On these servers reasoning tokens and answer tokens share one output budget, so a model that deliberates extensively can spend the whole budget before writing any code. That is what happened to Nemotron's deck; it was re-run with reasoning off, which is recorded on its row.
- —A judge cannot open a file. Every format it is asked to score has to be described to it — an earlier version of this harness passed only cell values, so a well-formatted workbook read as a bare data dump and scored 1.5 for all three. The extractor now reports bold cells, fills, borders, column widths, freeze panes, number formats and formula counts. A quality score is only as good as what the judge is shown.
- —The annual report is the one artifact where length is part of the brief — eight to twelve pages. That check now counts words against the page target rather than paragraphs, because a paragraph count is not a page count: a model can write fifty short paragraphs and still be three pages in. Qwen's report runs 1,985 words (~5.7 pages) and Nemotron's 1,478 (~4.2), so both fail on length. DeepSeek's is 3,815 words (~10.9 pages) and passes. Correcting the check moved Qwen's finale from four passes to three; nothing in the core suite moved.