What a task looks like.
550 tasks in total. Below is one real task from each of the six kinds of work, shown the way a model receives it — and the answer it has to produce to be graded correct.
Inbox triage and reply
A customer email arrives. Decide which queue it belongs to and what the company policy says should happen to it.
Reading intent from messy, real-sounding email
Applying a written routing guide
Choosing the right outcome from twelve policy outcomes
Hello, my customer reference is C0778 and I raised this on 2024-09-18. I need to change the billing address on my account, since the one you have on file is no longer where I'm living. Could you update it for me? Just tell me what you need from my end to get it done.
{ "queue": "Account management", "outcome": "phone_confirmation_required" }Billing address changes need a phone confirmation before they are applied.
Both answers compared against the key. Correct on both counts scores 1. Right on one of the two scores 0.5 — it is graded as wrong overall, but the partial credit is recorded.
Document extraction
A scanned invoice, purchase order or contract arrives in a range of layouts. Pull every field out of it, in one exact shape.
Finding the same fields across different layouts
Normalising dates and money
Keeping line items and totals attached to the right record
Extract the fields from invoice INV-2024-000114: invoice number, invoice date, due date, customer ID, and every line item with its SKU, quantity and amount. Dates as YYYY-MM-DD, amounts as plain numbers in dollars.
{ "invoice_number": "INV-2024-000114", "invoice_date": "2024-11-20", "due_date": "2024-12-20", "customer_id": "C0028", "line_items": [ { "sku": "…", "quantity": 13, "amount_cents": 14781 } ] }Dates normalised, amounts in cents, line items kept as rows.
Field by field against the generated document. Dates must match as dates, money as cents, and line items as rows — so 1,216.90 in the wrong place does not pass.
Spreadsheet analysis
Four company files are handed over — a sales ledger, a product catalogue and the customer list. Answer one question that requires combining them.
Multi-file joins and period filters
Excluding cancelled rows and applying the stated definitions
Returning the answer in the exact shape asked for
Among customers with at least $25,000 of revenue in both H1 2025 (January to June) and H2 2025 (July to December), which 5 had the largest fall in gross margin % from H1 to H2? Give their customer IDs, and explain what drove the fall.
{ "answer": ["C0805", "C0029", "C1024", "C0520", "C0761"], "explanation": "…" }Five IDs, computed from the ledger after excluding cancelled orders. This is one of the tasks the model missed.
Compared to the answer computed from the same files. Money is allowed 0.5% relative tolerance and one cent absolute; counts must match exactly; percentages must be within 0.1 of a point.
Formulas and SQL
Write a single SQL query or spreadsheet formula that answers a business question against the company database.
Correct joins, grouping and ordering
Getting the arithmetic definition right — list price versus discounted
Producing a query that actually runs
Which account reps look after more than 30 customers? Count customers per rep and give each rep's delivered revenue in 2024 in dollars. Order by customer count, highest first, then employee id. Columns: employee_id, customer_count, revenue_dollars.
SELECT e.id, COUNT(DISTINCT c.id) AS customer_count, ROUND(SUM(l.quantity * l.unit_price_cents) / 100.0, 2) AS revenue_dollars …
The query is run against the real database and its rows compared to the reference query's rows.
The query is actually run against the company database, and its rows are compared to the reference query's rows. Row order matters only when the question asked for an order. A query that errors scores zero.
Business writing
Read a customer email and the relevant policy, then write the company's reply — the one the customer would actually receive.
Deciding the outcome the policy gives
Citing the right policy section
Including the reference number in the body
Not promising anything the policy forbids
Read against Acme's policy, then draft the reply the customer would actually receive.
I'm writing on 2025-12-05 to ask for a new contact person to be added to our account, customer reference C0242. Can you tell me what details you need from me to make that change?
{ "outcome": "updated", "policy_section": "7.1", "reference": "C0242" }Graded on the outcome, the policy section cited, the reference inside the email, and no forbidden promise.
Four separate checks, all of which must pass: the outcome the policy gives, the correct policy section, the reference number appearing inside the email body the customer would read, and no promise the policy forbids for that case (for example offering a refund when a restocking fee applies). Prose quality is not scored by a model — the draft either states the right decision with the right citation and no forbidden promise, or it does not.
Classification and routing
An inbound sales lead arrives. Apply the routing rules in order and send it to the right account rep.
Rule ordering — first match wins
Reading country, size band and industry correctly
Handling leads that match no rule
Route this inbound sales lead to the right account rep. Apply the routing rules in order; the first rule that matches decides.
{ "rep_id": "E017", "rule": 1 }Rule 1 matched: country in (AE, SA, QA, OM) and size band 250+.
Exact match against the key.
Same tasks for every model.
A model runs all 550 tasks, three times each. Nothing is swapped out between models.
See the results