AI in the organisation
Less “AI strategy.” More “which jobs, which data, who owns it, and what you will measure.”
Five rabbit holes. None of them is a strategy slide.
Eighty-five minutes, five sections. The order is deliberate. First the evidence, because every decision that follows depends on being honest about how much value is actually showing up and where. Then jobs, because it is the question staff are asking whether or not leadership is. Then the operating model: who owns this, and what a 20-person firm should do differently from a 2,000-person one. Then measurement and money, which is where most initiatives quietly die. Then five things to do on Monday. Each section closes with four discussion questions.
Where the value is
Individuals get faster. Companies mostly do not, yet. The difference is adoption, workflow redesign and measurement, not the model.
Task gains · firm gapsSection one is the evidence, read as a skeptic would. The headline numbers from the big surveys, what the famous “95% fail” report actually measured, the controlled studies that do show gains, and the mechanism that explains why those two pictures are both true.
Most companies cannot yet see it in the P&L.
of organisations attribute any EBIT impact to AI. Unchanged from 2025, while nearly 9 in 10 use AI somewhere and 80% of individual users say it helps them personally.
are “high performers”: 5% or more of EBIT from AI and a “significant” impact. Also unchanged. Three-quarters of them redesigned workflows; a quarter of everyone else did.
of 300 enterprise GenAI deployments showed no measurable P&L impact, per MIT NANDA. Widely quoted, widely misread. Next slide.
Translation: nearly everyone is using it, most people like it, and most finance directors cannot find it in the numbers. All three are true at once.
Start with the numbers a board would ask for, and start by admitting what they say. McKinsey’s State of AI survey, published 25 August 2026 from 1,719 respondents in 97 countries, fielded May to June 2026, finds that nearly nine in ten organisations use AI regularly in at least one business function, and 44% say it is scaling across the enterprise. Eighty percent of people who use it in their own role say it has improved their personal productivity. And yet only 37% of respondents attribute any EBIT impact at all to AI, a figure essentially unchanged from the previous year, and only about 6% meet McKinsey’s definition of a high performer: at least 5% of EBIT attributable to AI and a self-described significant impact. That 6% has not moved either. McKinsey’s own line is that “organisations’ conviction in AI is growing faster than the immediate financial returns they can attribute to it”. The third number is the one everyone has heard. MIT’s NANDA initiative reported in August 2025 that 95% of the enterprise generative AI deployments it looked at showed no measurable profit-and-loss impact. It is a real finding from a real report, and it is also the most over-interpreted statistic of the last two years, so it gets its own slide. The one detail from McKinsey to hold on to for later: among the high performers, nearly three-quarters say they fundamentally redesigned workflows because of AI. Among everyone else, about a quarter. That difference is the subject of this section. Note that all these figures are self-reported by executives, not audited; treat them as a survey of belief as much as of results.
Sources (3)
The “95% fail” report. What it measured, and what it didn’t.
What it is
“The GenAI Divide: State of AI in Business 2025”, MIT NANDA, Aug 2025. About 150 executive interviews, 350 employee surveys and 300 public deployments.
What it found
About 5% of pilots reached rapid revenue acceleration; the rest showed “no measurable P&L impact”. Vendor tools succeeded ~67% of the time; internal builds a third as often.
What it blamed
A “learning gap”: generic tools that do not learn the workflow, and organisations that bolt a chatbot on instead of redesigning the process. Not the models, not regulation.
What it cannot tell you
“No measurable impact” often means nobody measured. The sample was not random. Executives described success in their own words. It says nothing about your company.
How to quote it
Say: “most pilots in one 2025 study had no measured P&L effect, mostly because they were never set up to be measured.”
Don’t say: “95% of AI projects fail.” That is not what it says, and the people who quote it that way did not read it.
Use it for: the two findings that do survive: buy before you build, and put it in the back office first.
The report’s real lesson is the one nobody quotes: the pilots that worked had a process owner, a workflow change and a number.
In August 2025 MIT’s NANDA initiative published “The GenAI Divide: State of AI in Business 2025”. Fortune ran it under the headline “95% of generative AI pilots at companies are failing” and the figure went round the world, briefly moving tech stocks. What the report actually did: roughly 150 interviews with executives, a survey of 350 employees, and an analysis of 300 public AI deployments. What it found: about 5% of pilots achieved rapid revenue acceleration; the rest, in the report’s words, showed no measurable P&L impact. It attributed this to a “learning gap”, meaning generic tools that do not adapt to a company’s workflows, and organisations that bolt a chatbot on rather than changing how work is done. It explicitly did not blame model quality or regulation. Two secondary findings are more useful than the headline. Tools bought from specialised vendors succeeded about two-thirds of the time; internal builds succeeded about a third as often. And the biggest returns were in back-office automation, while more than half of budgets went to sales and marketing. Now the caveats, which the report’s fans skip. “No measurable impact” is not the same as “no impact”: most pilots had no baseline, so there was nothing to measure against. The 300 deployments were the ones the researchers could see, not a random sample. The interviews captured executives’ accounts of success. Fortune itself noted that NANDA, which works on agent protocols, might have an institutional interest in current approaches looking inadequate, while finding no evidence of skewing. So quote it precisely: in one 2025 study, most enterprise pilots had no measured P&L effect, largely because they were never designed to produce one. That is a finding about measurement and workflow design, and it is the right finding.
Where the gains are real: four controlled studies.
| Study | Who and what | Result | The detail that matters |
|---|---|---|---|
| Brynjolfsson, Li & Raymond · 2023, QJE 2025 | 5,179 support agents at one firm; AI assistant suggesting replies | +14% issues resolved per hour | Novices +34%; experienced staff roughly nil. Better customer sentiment, lower attrition. |
| Noy & Zhang · Science, Jul 2023 | 453 college-educated professionals; writing tasks with ChatGPT | 40% faster, quality +18% | The weakest writers gained most. Gap between workers shrank. |
| Cui et al. · Microsoft, Accenture + a Fortune 100 · 2024 | ~4,900 developers in three field trials; GitHub Copilot | +26% completed tasks | Juniors +27–39%; seniors +8–13%. Juniors adopted more. |
| METR · Jul 2025 | 16 expert open-source developers, 246 tasks, own mature codebases | 19% slower with AI | They predicted +24% and believed +20% afterwards. Familiar code, little documentation. |
Pattern: large gains on bounded tasks, largest for the least experienced, smallest or negative for experts on work they already know cold. Self-reports overstate it.
Against the survey gloom, the controlled evidence of gains is solid, as long as you read what it covers. Brynjolfsson, Li and Raymond studied the staggered rollout of a generative AI assistant to 5,179 customer support agents at one software company; issues resolved per hour rose 14% on average, with novice and low-skill agents improving by 34% and experienced, high-skill agents barely at all. Customer sentiment improved and attrition fell. The paper was published in the Quarterly Journal of Economics in 2025. Noy and Zhang, in Science in July 2023, gave 453 college-educated professionals realistic writing tasks; with ChatGPT they finished about 40% faster and independent graders scored the output 18% higher, and the gap between the best and worst writers narrowed. Cui and colleagues ran three field experiments with GitHub Copilot at Microsoft, Accenture and an unnamed Fortune 100 manufacturer, around 4,900 developers in total; completed tasks rose 26% on average, with recent hires and juniors gaining 27 to 39% and seniors 8 to 13%. Then the counterexample. METR, in July 2025, ran a randomised trial with 16 very experienced open-source developers working on their own large, mature codebases; with AI tools allowed they took 19% longer. They had predicted a 24% speed-up and, after the study, still believed they had been 20% faster. The authors are careful: the setting is specific, and the result does not mean AI is useless elsewhere. Read together, the four studies say one consistent thing. Gains are real and large on bounded, well-specified tasks; they are largest for people who are newer or weaker at the task; they shrink or reverse for experts on work they already know intimately; and people’s own estimates of the benefit are unreliable, which is why Section 04 insists on a baseline.
Four leaks between a faster task and the P&L.
Task gets faster
A person with the tool drafts, summarises or codes 20–40% faster. The studies say so.
Leak 1 · Adoption
Half the team never uses it, or uses it on personal accounts nobody sees. Microsoft, 2024: 78% of AI users brought their own tool.
Leak 2 · The workflow
The step got faster. The approval chain, the hand-offs and the queue did not. Saved minutes become slack, not output.
Leak 3 · Measurement
No baseline, so “hours saved” is a feeling. The MIT 95% is mostly this leak.
Leak 4 · The P&L
Even measured gains land as “capacity”. Unless someone redeploys the time or the headcount, EBIT does not move.
A faster step in an unchanged process is a coffee break. Value needs the process to change around the step.
Here is the mechanism that makes both the task studies and the firm surveys true. A person with the tool gets faster at a bounded task; that part is established. Between that and a line in the accounts there are four places the gain leaks out. The first is adoption. In most organisations half the team never really uses the tool, or uses a personal account that the company cannot see, measure or secure. Microsoft’s 2024 Work Trend Index, from 31,000 knowledge workers, found 75% using generative AI at work and 78% of those bringing their own tools. The second is the workflow. A step got faster; the approval chain, the hand-offs and the queue it sits in did not. Ten minutes saved on a draft that then waits two days for sign-off is ten minutes of slack. The third is measurement. Without a baseline, “hours saved” is a feeling, and the controlled studies show that feelings about AI productivity are unreliable in both directions. Most of the MIT 95% is this leak. The fourth is the P&L itself. Even a measured, real saving lands as capacity: people have more time. Unless someone decides what the time is for, whether that is more cases, faster cycle times, fewer contractors or not back-filling a vacancy, the earnings line does not move. McKinsey’s high performers are not using better models than everyone else; they have redesigned workflows, which addresses the second and fourth leaks, and they measure, which addresses the third. Adoption, the first leak, tends to follow once the tool is inside the process rather than a tab beside it. The rest of this deck is, in effect, about plugging the four leaks in order.
The four places it pays first. And the places it doesn’t.
1 · Messy input, high volume
Inboxes, invoices, claims, tickets, forms that a person reads all day. Part III’s exception queues. The model reads; code checks; a human approves. Measured in cost per case.
2 · Drafting and summarising inside existing roles
Proposals, reports, meeting notes, first drafts of anything. The Noy & Zhang result. Cheap to switch on; value depends on whether the time is redeployed.
3 · Customer support with a human path
Routine questions resolved; anything angry, complex or high-value handed to a person. The Brynjolfsson result, and the Klarna lesson. Measured in resolution, sentiment and escalation.
4 · Developer productivity
Part XI. +26% in field trials for most developers; expert-on-familiar-code can go the other way. Measured in completed work and defect rates, never lines of code.
Pick by volume, messiness and reversibility. Not by which department shouted loudest.
Four places pay first, and they are the same four in nearly every company this series has looked at. First, messy input at volume: the inboxes, invoices, claims, tickets and forms that somebody reads all day, and especially the exception queues that Part III identified as the best first agent project. The model reads, code checks the arithmetic and the rules, a human approves anything that matters, and the metric is cost and time per case. Second, drafting and summarising inside existing roles: proposals, reports, minutes, first drafts. This is the Noy and Zhang result and it is the cheapest to switch on, because the tools your people already pay for do it; the catch is that the value only exists if the saved time is redeployed into something. Third, customer support with a human path: routine questions handled, everything angry, complex or high-value escalated to a person quickly. This is the Brynjolfsson result and the Klarna lesson in one, and the metrics are resolution rate, customer sentiment and how fast a person can be reached. Fourth, developer productivity, covered in Part XI: a 26% lift in completed tasks across three field trials for ordinary developers, with the warning from METR that experts on code they know intimately may get slower. Measure finished work and defects, never lines of code. Where it does not pay yet is just as consistent. Rare processes give you no volume to learn from or to spread the setup cost over. High-stakes, irreversible decisions, money leaving, legal and medical judgement, decisions about people, carry Part VIII’s regulatory weight and the full cost of a confident wrong answer. And a process nobody can explain cannot be automated; the agent will faithfully automate the disagreement.
Sources (5)
- https://www.nber.org/papers/w31161
- https://news.mit.edu/2023/study-finds-chatgpt-boosts-worker-productivity-writing-0714
- https://mitsloan.mit.edu/ideas-made-to-matter/how-generative-ai-affects-highly-skilled-workers
- https://getdx.com/blog/metr-study-on-how-ai-affects-developer-productivity/
- https://www.entrepreneur.com/business-news/klarna-ceo-reverses-course-by-hiring-more-humans-not-ai/491396
Value against risk. Start top-left; earn your way right.
Do first
High volume, mistakes cheap and reversible. Triage, drafting, summarising, internal search, first-pass extraction. Draft mode by default; promote per action.
Do carefully
High volume, mistakes expensive. Customer replies, refunds, invoice approval. Act-within-limits with hard caps, a test set, a human path and monthly trace sampling.
Do if cheap
Low volume, mistakes cheap. Personal assistants, one-off research, meeting notes. Switch on what you pay for; do not build anything.
Don’t, yet
Low volume, mistakes expensive. Contracts, hiring decisions, medical or legal advice, anything about one person. Suggest mode at most; a human owns every outcome.
Volume pays for the setup. Reversibility pays for the mistakes. You need at least one of them, and your first project needs both.
A two-by-two to carry round, with volume on one axis and the cost of a mistake on the other. Volume matters because every AI deployment has a setup cost, a test set, an owner, a workflow change, and only volume spreads it thin enough to pay back. Reversibility matters because the model will sometimes be confidently wrong, and the question is whether that costs you a correction or a customer. Top-left, high volume and cheap mistakes, is where to start: triage, drafting, summarising, internal search, first-pass extraction from documents. Run it in draft mode and promote individual actions as the log earns it, exactly as Part III described. Top-right, high volume and expensive mistakes, is where the money is and where the discipline has to be: customer replies, refunds, invoice approvals. Act within hard-coded limits, keep a test set, guarantee a human path and sample the traces monthly. Bottom-left, low volume and cheap mistakes, is the personal-assistant zone: switch on the tools you already pay for and do not build anything. Bottom-right, low volume and expensive, irreversible mistakes, is where the hype goes to die: contracts, hiring decisions, medical or legal advice, anything that decides something about one person. At most the model suggests, a human owns the outcome, and Part VIII’s regulatory map applies. The rule of thumb underneath: volume pays for the setup and reversibility pays for the mistakes. A project needs at least one; your first project needs both.
Where would your finance director find it?
- Which of the four leaks is biggest in your company today: adoption, workflow, measurement or redeployment?
- Name one process with high volume and cheap, reversible mistakes. Who reads its input all day?
- If every “hours saved” claim from the last year were true, where did the hours go?
- Which AI tool in use has a baseline measurement from before it arrived? Any?
The third question is the uncomfortable one. If the hours were saved and nothing changed, the saving was a coffee break. That is not a criticism of the staff; it is a criticism of the plan.
Jobs and people
Tasks change before jobs do. The early signal is at the bottom of the ladder, not the top. Say what you know; say what you don’t.
Tasks before jobsSection two is the question staff are asking whether or not leadership is. The evidence is genuinely mixed and this section tries to show it as it is: no economy-wide displacement yet, a real and growing gap for the youngest workers in exposed jobs, and a rising number of employers naming AI in layoff announcements.
Three numbers, three caveats. Autumn 2026.
Employment gap for 22–25 year-olds in the most AI-exposed occupations versus peers in less-exposed ones, as of June 2026. Up from 15% a year earlier. Mostly reduced hiring, not firing. Experienced workers: no gap.
How far the US occupational mix has moved beyond the internet-era path in the 33 months after ChatGPT. “No discernible disruption” economy-wide. Descriptive, not a forecast.
US job cuts in which employers cited AI, Jan–Aug 2026, of 529,914 announced (22%). AI was the top stated reason for five months running. A stated reason, not a verified cause.
No wave yet. A clear early signal for juniors in exposed jobs. And a growing habit of blaming AI in layoff press releases, which is not the same thing as AI doing the work.
Three numbers, each with a caveat built in. First, Stanford’s Digital Economy Lab, in “Canaries in the Coal Mine” by Brynjolfsson, Chandar and Chen, uses ADP payroll data covering millions of US workers. The August 2026 update finds that employment for 22 to 25 year-olds in the most AI-exposed occupations is about 19% below where it would be had it tracked similarly aged workers in less-exposed occupations, up from 15% at the July 2025 data vintage. The gap comes mainly from reduced hiring rather than increased separations, is concentrated where AI automates rather than complements tasks, and does not appear for experienced workers. The authors call these descriptive patterns, not causal estimates, and say they cannot yet answer the causal question definitively. Second, the Yale Budget Lab with Brookings, in October 2025, built an index of how much the US occupational mix has shifted and found that 33 months after ChatGPT’s release the labour market showed no discernible disruption: the change was only about one percentage point beyond the path seen during the internet’s rise. They note that technological disruption historically takes decades, that their analysis is not predictive, and that it is too soon to tell. Third, the outplacement firm Challenger, Gray and Christmas counts announced US job cuts by the employer’s stated reason. Through August 2026, employers cited AI in 116,175 of 529,914 announced cuts, and AI was the leading monthly reason from March to July. That is a stated reason in a press release; Challenger does not verify that AI did the work, and “AI” has become a respectable thing to say when cutting costs. Put together: no economy-wide wave yet, a real early signal at the bottom of the ladder, and a growing number of companies invoking AI whether or not it is the cause.
Tasks change before jobs do. That is where to look.
Augmentation, mostly. So far.
- A job is a bundle of tasks. AI takes some tasks out of some bundles; the bundle gets re-cut before the job disappears.
- The biggest measured gains go to the least experienced: the tool carries the knowledge the junior hasn’t got yet (Brynjolfsson; Cui).
- Where AI complements a role, employment is flat or rising, especially for experienced workers (Stanford, 2026).
- McKinsey, Aug 2026: 39% expect AI to cut their headcount next year. Only 14% say it actually did last year. Expectations keep running ahead.
Not nothing. Not half the jobs.
- Firms that cut first and measured later have reversed: Klarna rehired support staff; CBA reinstated 45 roles after call volumes rose.
- The junior gap is real and growing: if the tool does the tasks juniors learned on, nobody learns. Next slide.
- Headlines count announcements that cite AI. Nobody audits whether AI did the work.
- Every number here is 2023–2026 US data on the first generation of tools. It will date. Re-check before you repeat it.
Plan for task change in every role this year. Plan for job change in a few. Announce neither until you have measured something.
The useful unit is the task, not the job. A job is a bundle of tasks, and what the controlled studies measure is AI taking some tasks out of some bundles: the first draft, the summary, the routine reply, the boilerplate code. Bundles get re-cut long before jobs disappear, which is why the firm-level data is so much quieter than the task-level data. Three things the evidence supports. The largest measured gains go to the least experienced, because the tool carries the knowledge the junior has not yet accumulated; that was true in the support study and in the developer trials. Where AI complements a role rather than automating it, employment is flat or rising, particularly for experienced workers, per the 2026 Stanford update. And expectations keep running ahead of events: in McKinsey’s August 2026 survey 39% expected AI to reduce their organisation’s headcount in the coming year, up from 32% a year before, while only 14% of AI-using organisations said it had actually contributed to an overall decline in the year just gone, less than half of what the previous survey had predicted. Three things it does not support. It does not support the comfortable view that nothing is happening, because the junior gap is real and growing. It does not support cutting first and measuring later: Klarna went back to hiring support staff, and the Commonwealth Bank of Australia reinstated 45 roles within weeks when the voice bot that was supposed to replace them coincided with rising call volumes. And it does not support treating layoff announcements as evidence; nobody audits whether AI did the work. Every number on these slides is US data from the first generation of tools, 2023 to 2026. It will date.
Sources (6)
- https://digitaleconomy.stanford.edu/news/canariesaug26/
- https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
- https://www.entrepreneur.com/business-news/klarna-ceo-reverses-course-by-hiring-more-humans-not-ai/491396
- https://www.abc.net.au/news/2025-08-21/cba-backtracks-on-ai-job-cuts-as-chatbot-lifts-call-volumes/105679492
- https://www.nber.org/papers/w31161
- https://mitsloan.mit.edu/ideas-made-to-matter/how-generative-ai-affects-highly-skilled-workers
The junior pipeline. Nobody owns it, so it happens.
The mechanism
The tasks AI does best are the tasks juniors used to learn on: first drafts, first-pass research, routine tickets, boilerplate code. Remove them and the junior is cheaper to not hire. Stanford: the gap is mostly fewer hires, not more exits.
The bill, later
Your seniors came up through those tasks. In five years you need people who can judge the model’s output, and judgement is learned by doing the work badly first, with supervision. No juniors now means no seniors then.
What a company can do
Hire juniors into redesigned roles: reviewer, exception handler, test-set builder. Make “check the AI’s draft, explain what you changed” a graded task. Keep one unassisted lane per role. Measure junior progression, not just output.
If AI does all the junior work, your senior pipeline is being quietly switched off. Nobody will announce it.
The one labour-market finding with a clear signal is also the one with a clear mechanism, and it is worth spelling out because no individual decision looks wrong. The tasks AI does best, first drafts, first-pass research, routine tickets, boilerplate code, are exactly the tasks on which junior staff used to learn the job. Remove those tasks and the junior is no longer needed this quarter, so the vacancy is not back-filled. The Stanford data says this is how it shows up: fewer hires of 22 to 25 year-olds into exposed roles, not more dismissals. Every one of those decisions is locally sensible. The bill arrives later. Your senior people came up through those tasks; that is where they learned to tell a right answer from a plausible one. In five years you will need people who can judge model output, build test sets and handle exceptions, and judgement is learned by doing the work badly first under supervision. No juniors now means no seniors then, and the gap will not be visible in any quarterly number. What a company can do is concrete. Hire juniors into roles that have been redesigned around the tool: reviewer, exception handler, test-set builder, the person who checks the draft and explains what they changed. Make that checking a graded, supervised task rather than a chore. Keep one unassisted lane per role so the underlying skill is still practised. And measure junior progression as a metric alongside output. This is not altruism. Part III’s edit rate only works if someone capable of editing exists; Part VII’s evals only work if someone can recognise a wrong answer. Both are skills, and both are built in the first years of a career.
Tell your people what you know, and what you don’t.
Which tasks will change this year, specifically
“Drafting proposals, first-pass invoice checks, tier-one tickets.” Named tasks, named teams. Not “AI will transform how we work.”
What you are measuring, and that you will share it
The baseline, the metric and the pilot result. If the number is bad, say so. People can smell a hidden spreadsheet.
What happens to time that gets freed
Decide before the pilot: more cases, faster cycle time, fewer contractors, no back-fill. Then say it. Silence is read as “layoffs”.
What you do not know
“We don’t know what this does to headcount in three years. Nobody does. We will tell you what we learn each quarter.” It is the only credible sentence.
Why this matters
57% of employees hide their AI use at work; 66% rely on its output without checking it. KPMG / University of Melbourne, 48,000 people, 47 countries, Nov 2024–Jan 2025.
Only 26% of AI users say their leadership is consistently aligned on AI; 65% fear falling behind. Microsoft Work Trend Index, May 2026.
People who think they are being replaced hide what they know. That is the one outcome you cannot afford.
The honest version is also the cheapest. Everything else gets found out, and then nothing you say about AI is believed.
Most companies say nothing to staff about AI and jobs, or say “AI will not replace people, it will augment them”, which staff correctly translate as “they have not decided yet”. Four things to say instead, and one to admit. Say which tasks will change this year, specifically: drafting proposals, first-pass invoice checks, tier-one tickets, named teams. Say what you are measuring and that you will share it, including when it is bad; a hidden spreadsheet is the thing people fear most. Say what will happen to the time that gets freed, and decide this before the pilot rather than after: more cases, faster cycle time, fewer contractors, not back-filling a vacancy. Silence on this point is read as layoffs. And say what you do not know: nobody can tell you what this does to headcount in three years, so say that, and commit to reporting what you learn each quarter. It is the only sentence that is both true and believable. The panel shows why this matters. The KPMG and University of Melbourne study of over 48,000 people in 47 countries, fielded November 2024 to January 2025, found 57% of employees hiding their AI use and presenting AI output as their own, 66% relying on output without checking its accuracy, and only 40% saying their workplace has any policy or guidance on generative AI. Microsoft’s 2026 Work Trend Index, from 20,000 AI-using workers, found only 26% saying leadership is consistently aligned on AI and 65% fearing they will fall behind. People who believe they are being replaced hide what they know, which is precisely the knowledge you need to redesign the process. The honest version is the cheap version.
Sources (3)
- https://fbe.unimelb.edu.au/newsroom/media-release-global-study-reveals-trust-of-ai-remains-a-critical-challenge-reflecting-tension-between-benefits-and-risks
- https://finance.yahoo.com/sectors/technology/articles/microsoft-finds-just-13-firms-044659301.html
- https://www.forbes.com/sites/moorinsights/2026/05/19/microsoft-work-trend-index-2026-shows-ai-productivity-is-not-enough/
Who in your company is learning the job right now?
- Which roles did you stop hiring juniors into in the last two years? Was that a decision, or did it just happen?
- If a 24-year-old joined your team on Monday, which tasks would they learn on, and does the tool now do them?
- What has leadership actually said to staff about AI and jobs? What did staff hear?
- When the pilot frees up time, what is it for? Has anyone decided?
The first question usually gets an honest answer: it just happened. That is the junior-pipeline problem in one sentence.
The operating model
Not IT alone. Not a chief AI officer by default. A list of uses, a name against each, a small group that enables, owners who are measured.
Owners, not committeesSection three: who owns this, what the inventory looks like, what a 20-person, 200-person and 2,000-person company should realistically do, and how to buy the tools without giving your data away.
Who owns AI. Three roles, none of them new.
Owners in the business
- One named person per use, as in Part VIII. The head of the team whose process it is.
- Owns the baseline, the metric, the test set and the decision to scale or stop.
- Is measured on the process outcome, not on “AI adoption”.
- Signs off each step up the autonomy dial.
A small central enabling group
- Two to five people in a 2,000-person firm; one person part-time in a 200-person firm; nobody in a 20-person firm.
- Procures tools and negotiates data terms once. Keeps the inventory. Runs the policy.
- Builds shared plumbing: connectors, evals, logging (Part XI).
- Does not own use cases. Helps owners who do.
Champions in every team
- Volunteers who already use the tools well. Usually not managers.
- Given time: a few hours a week, visibly protected.
- Show colleagues what works in their actual job; feed the central group what breaks.
- The cheapest change-management budget line you will ever approve.
IT keeps the keys and the security. A chief AI officer is a fine idea at 5,000 people and a vanity hire at 200. The owner in the business is the role that cannot be skipped.
Who owns AI in a company is the question that produces the most org-chart theatre and the least value, so here is the plain answer: three roles, none of them new. The first and the only indispensable one is an owner in the business for each use, the same person Part VIII put against each line of the inventory. Usually the head of the team whose process it is. They own the baseline, the metric, the test set, and the decision to scale or stop; they are measured on the process outcome, not on adoption percentages; and they sign off every step up the autonomy dial from Part III. The second is a small central enabling group. In a 2,000-person company that is two to five people; in a 200-person company it is one person part-time; in a 20-person company it is nobody, or the founder on Friday afternoons. The group procures tools and negotiates data terms once for everybody, keeps the inventory, runs the policy, and builds the shared plumbing that Part XI described, connectors, evaluation harnesses and logging. What it must not do is own use cases; the moment it does, the business stops owning them. The third is champions: volunteers in every team who already use the tools well, often not managers, given a few visibly protected hours a week to show colleagues what works in their real job and to tell the central group what breaks. It is the cheapest change-management budget line you will approve. IT keeps identity, keys and security, as it does for every other system. A chief AI officer is reasonable at 5,000 people with many regulated uses; at 200 it is a title in search of a job. The role that cannot be skipped is the owner in the business.
The inventory. One row per use, one name per row.
| Use | Owner | Reads | May do | Plan / tier | Autonomy | Metric |
|---|---|---|---|---|---|---|
| Support triage (Part III) | Head of Support | Tickets, CRM, policy docs | Tag, draft reply, refund ≤ $50 | Vendor, business tier | Act within limits | Cost per ticket; CSAT |
| Proposal drafting | Sales Director | Past proposals, CRM notes | Draft only | ChatGPT Business | Draft | Hours per proposal; win rate |
| Invoice extraction | Finance Manager | AP inbox, ERP read-only | Extract, explain mismatches | API via n8n | Draft | Exceptions cleared per day |
| Coding assistant | Eng. Lead | Repos | Suggest code in editor | Copilot, per seat | Draft | PRs merged; defect rate |
| “We use ChatGPT for that” | Nobody, yet | Unknown | Unknown | Personal accounts | Unknown | None |
If the inventory fits on one page, you can govern it, budget it and measure it. If it does not exist, you are doing all three by rumour.
Part VIII put the inventory first for governance. It is also the first thing you need for budgeting, for measurement and for deciding what to do next, which is why it appears again here. The shape is one row per use and one named person per row. For each use: what it reads, what it may do, which plan or tier it runs on, where it sits on the autonomy dial, and the single metric its owner is measured on. The rows on the slide are invented but typical. A support triage agent of the kind Part III built, owned by the head of support, reading tickets and policy, allowed to tag, draft and refund under a cap, measured on cost per ticket and satisfaction. Proposal drafting on a business-tier chat tool, draft mode only, measured on hours per proposal and win rate. Invoice extraction built on a workflow tool against the API, measured on exceptions cleared per day. A coding assistant on per-seat licences, measured on merged pull requests and defect rate. And the last row, which is the one you will actually find when you ask: “we use ChatGPT for that”, on personal accounts, owned by nobody, reading who-knows-what, measured on nothing. The KPMG study says 57% of employees hide their AI use; the Netskope 2026 report, from network telemetry, found 47% of workplace generative AI users still on personal accounts, down from 78% a year earlier as companies bought business tiers. The purpose of the inventory is to turn the last row into one of the others. If it fits on a page you can govern it, budget it and measure it. If it does not exist, you are doing all three by rumour.
Build, buy or switch on. By the size of the company.
| 20 people | 200 people | 2,000 people | |
|---|---|---|---|
| Default move | Switch on what you already pay for. Business tier of one chat tool for everyone. Nothing else for six months. | Business tiers for all. One lighthouse process on a workflow tool or a vendor’s agent feature. Buy before build. | Business tiers; platform contract with API access; a central group building shared plumbing; two or three owned lighthouse processes. |
| Who owns it | The founder, Friday afternoons. One champion. | One manager part-time as the enabling group; an owner per process; champions in each team. | A small central team (2–5); owners in each business unit; a champion network; IT for keys and security. |
| When to build (Part XI) | Almost never. Glue with n8n/Zapier at most. | When a vendor cannot reach your data, or the process is your edge. One developer-quarter, with evals. | When volume justifies it and the process is core. In-house with the SDKs; buy everything peripheral. |
| When to hire | Don’t. Contract a day of help to set up the policy and the tiers. | One person who can build workflows and run evals, if the lighthouse works. Not a “head of AI”. | Engineers who can build agents with leashes; an evals specialist; a procurement lead who reads data terms. |
| Budget shape | Seats. Under $10k a year all-in. | Seats + one build + people’s time. Low six figures, licences a minority of it. | Platform + seats + team. Licences are the small line; integration and data are the big ones. |
At every size the default is the same: switch on what you pay for, measure one process, and buy before you build.
What a company should do depends more on its size than on its sector, so here is the same question answered three times. A 20-person firm should switch on the business tier of one chat tool for everyone, name the founder as owner and one keen person as champion, and do nothing else for six months while it learns what people actually use it for. It should almost never build; a workflow tool for glue is the limit; and it should not hire, but may usefully pay for a day of outside help to set up the policy and the tiers. All-in, under ten thousand dollars a year. A 200-person firm should do the same with tiers, then pick one lighthouse process from the top-left of the value-risk quadrant and run it on a workflow tool or on the agent feature inside software it already pays for. One manager part-time plays the enabling group; one owner per process; champions in each team. It builds only when a vendor cannot reach its data or the process is its competitive edge, and then it budgets a developer-quarter with evaluations, not a weekend. If the lighthouse works, it hires one person who can build workflows and run evals; it does not hire a head of AI. The budget is low six figures with licences a minority of it. A 2,000-person firm adds a platform contract with API access, a central team of two to five building shared connectors, evaluation and logging, two or three owned lighthouse processes in different units, and a procurement lead who reads data terms. It builds in-house only where volume and strategic weight justify it, and buys everything peripheral. The MIT report’s finding that vendor tools succeeded about twice as often as internal builds is the reason “buy before build” appears in every column.
Buying it. Five questions before you sign anything.
Does it train on our data? Where does the data go?
Business, team and enterprise tiers: no by default. Consumer plans: often yes unless opted out. Part VIII’s table. Get it in the contract, not the brochure.
Which model is underneath, and can we swap it?
A vendor wrapper on a frontier model is fine; a vendor wrapper you cannot leave is not. Ask what happens to your prompts, data and logs on exit.
Can we see the trace and run our own test set?
Part III’s sniff test. If you cannot run your 100 past cases through it before buying, you are buying a demo.
What does it cost at our volume, not theirs?
Seats are predictable; token or credit billing is not. Model a normal month and a bad month. Ask for a cap.
Who at the vendor answers when it is wrong?
Air Canada owned its chatbot’s answer. You will own this one. A named contact, an SLA and an incident process, in writing.
The data-terms rule
Business tier for everyone who uses it. Then nobody has to remember the rule.
The cheapest control in the series, and the one most companies have still not bought. Netskope, 2026: 47% of workplace GenAI use is still on personal accounts, down from 78% a year earlier.
If the tool is a seat, buy the seat. If it is an agent, buy the trace.
Procurement is where governance becomes cheap: one negotiation, one data clause, one inventory row, and the rule enforces itself.
Most of the governance in Part VIII and most of the measurement in the next section can be made cheap at the point of purchase, so procurement deserves five questions of its own. First, does the vendor train on your data, and where does it go? Business, team and enterprise tiers from the major vendors do not train on your inputs by default; consumer plans often do unless opted out, as Part VIII’s table showed. Get the answer in the contract, not the brochure. Second, which model is underneath, and can you swap it? A vendor product wrapped around a frontier model is normal and fine; a product you cannot leave without losing your prompts, data and logs is not. Ask what exit looks like. Third, can you see the trace and run your own test set? This is Part III’s sniff test applied to buying: if you cannot push your hundred real past cases through the product before signing, you are buying a demo. Fourth, what does it cost at your volume? Seat licences are predictable; token and credit billing is not, and several vendors have moved to credits. Model a normal month and a bad month and ask for a cap. Fifth, who at the vendor answers when it is wrong? Air Canada was held liable for its chatbot’s invented policy, and you will be liable for this product’s; a named contact, a service level and an incident process belong in writing. The panel repeats the single cheapest control in the series: buy the business tier for everyone who uses the tools. Netskope’s 2026 report, based on network telemetry across its customers, found personal-account use of generative AI at work had fallen from 78% to 47% in a year as companies did exactly that. Half is still a lot.
Who owns the row?
- For each AI use you know about: who is the named owner, and what number are they measured on?
- Is there a central person or group that buys and sets terms, or does every team sign its own licence?
- Which column on the company-size table are you in, and are you behaving like it, or like the column to the right?
- What would it take to run your 100 past cases through the next tool before you buy it?
The third question catches the most common mistake: 200-person firms building like 2,000-person firms. The fourth is the one to take to the next vendor meeting.
Measuring and paying
A baseline, a metric, a pilot that is allowed to fail, a rollout that changes the workflow, and a budget where licences are the small line.
Baseline firstSection four is where initiatives live or die: what to measure and what not to, a scorecard, the pilot design, the company-level rollout path, change management without the incense, and what it actually costs.
What to measure. And what to stop pretending is a measure.
Things with a baseline.
- Time per case, measured by the system, not recalled by the person.
- Quality: a scored sample, judged by someone who did not produce it (Part VII).
- Error and rework rate: how often output was corrected after the fact.
- Cycle time end to end: did the customer get it sooner?
- Cost per case, all-in: licences, tokens, review time.
- Adoption: weekly active use inside the process; edit rate on drafts (Part III).
Things that only look like numbers.
- “Hours saved” from a survey with no baseline. METR’s developers believed +20%, measured −19%.
- Lines of code, words written, emails sent. Output volume is not value.
- Prompts, chats, “AI interactions”. Usage is a cost, not a result.
- Licences deployed. That is a procurement metric.
- Vendor-reported ROI. Never once negative.
- Anything self-reported by the team that gets the budget if the number is good.
If it was not measured before the tool arrived, it cannot be measured after. The baseline is the whole game.
Measurement is the leak that sinks most initiatives, so here is the list. On the left, six things worth measuring per process, all of which require a baseline taken before the tool arrives. Time per case, measured by the system rather than recalled by the person. Quality, as a scored sample judged by somebody other than the producer, which is Part VII’s evaluation discipline applied to a business process. Error and rework rate: how often the output had to be fixed afterwards, which catches the fast-but-wrong failure mode. Cycle time end to end, because a faster step in a slower chain is invisible to the customer. Cost per case, all in: licences, tokens and the human review time, which is where draft-mode savings often evaporate. And adoption, as weekly active use inside the process, together with the edit rate on drafts that Part III used to decide when an action had earned more autonomy. On the right, the things that look like numbers and are not. Hours saved from a survey with no baseline: METR’s expert developers believed they had been 20% faster while the stopwatch said 19% slower. Lines of code, words, emails: output volume. Prompts and chats: that is usage, which is a cost. Licences deployed: a procurement figure. Vendor-reported ROI, which has never once been negative. And anything self-reported by the team whose budget depends on the number being good. The rule that falls out is short. If it was not measured before the tool arrived, it cannot be measured after. The baseline is the whole game, and it is the step every failed pilot skipped.
A scorecard for one process. Fits on a page; decides at week 12.
| Metric | Baseline | Target | Week 4 | Week 12 | Read |
|---|---|---|---|---|---|
| Time per case (system clock) | 18 min | ≤ 12 min | 14 min | 11 min | Hit |
| Quality score (blind sample of 50, 1–5) | 3.9 | ≥ 3.9 | 3.6 | 4.0 | Dipped, recovered |
| Rework rate (corrected after send) | 6% | ≤ 6% | 11% | 5% | Watch the first month |
| Cycle time to customer (median) | 2.1 days | ≤ 1.5 days | 2.0 days | 1.9 days | Missed: approval queue |
| Cost per case, all-in | $9.40 | ≤ $7 | $9.10 | $6.80 | Hit, barely |
| Weekly active users / edit rate on drafts | — / — | ≥ 80% / falling | 55% / 70% | 85% / 35% | Hit |
Notice the quality dip and the rework spike in month one. They are normal. A pilot without a quality line would have called month one a success.
This is what a one-process scorecard looks like. The numbers are invented; the shape is what to copy. Six metrics in rows, matching the previous slide. A baseline column, taken from four weeks of system data before the tool was switched on. A target agreed before the pilot started, so that nobody can move the goalposts afterwards. Readings at week four and week twelve. And a plain-language reading of each line. Three things in the invented data are typical of real pilots. First, the quality score dips and the rework rate spikes in the first month, as people over-trust the draft and under-edit it, and then both recover as the edit rate settles; a pilot without a quality line would have called month one a success on time per case alone. Second, the cycle time to the customer barely moves even though time per case nearly halves, because the approval queue behind the step did not change; that is leak two from Section 01 showing up in the numbers, and the correct decision at week twelve is to fix the queue before scaling the tool. Third, adoption starts low and the edit rate starts high; what you want is both lines moving in the right direction, not a high score on day one. The last column matters as much as the numbers: at week twelve the owner reads the card and makes one of three calls, scale it, fix something specific first, or stop. A pilot that cannot end in “stop” is not a pilot.
The pilot that is allowed to fail. 30 to 90 days, one process.
Define success first
Owner named. One process. The metric, the target and the baseline written down before anyone logs in.
Baseline, 4 weeks
System-measured. If you cannot measure the process now, you cannot pilot it. Fix that first.
Run it, 30–90 days
Shadow then draft mode (Part III). Champions in the team. Weekly scorecard. Do not add a second process.
Read the card
Hit, missed, or hit for the wrong reason. Quality and rework count as much as speed.
Decide, out loud
Scale, fix one thing first, or stop. Tell the team which, and why. Stopping is a result, not a failure.
A pilot is an experiment, and an experiment that cannot fail is a rollout wearing a lab coat. The design is five steps, and the order is the point. Define success first: the owner is named, one process is chosen, and the metric, the target and the stop condition are written down before anyone logs in. Take a baseline for four weeks using system data; if the process cannot be measured today, that is the first project, not the AI. Run it for 30 to 90 days, starting in shadow mode and moving to draft mode as Part III described, with the team’s champions involved, a weekly scorecard, and a firm refusal to add a second process halfway through. Read the card honestly: hit, missed, or hit for the wrong reason, with quality and rework weighted as heavily as speed. Then decide out loud: scale, fix one specific thing first, or stop, and tell the team which and why. Stopping is a result; it tells you where value is not, which is worth knowing. The three failure modes at the bottom are the ones that recur. A pilot with no written stop condition cannot fail and therefore proves nothing. Three pilots at once with none measured is the most common pattern in companies with an enthusiastic sponsor, and it produces three anecdotes. And success declared by the team that gets the budget if the number is good is not a measurement; blind quality scoring by another team, with the baseline held by finance, costs little and removes the incentive. The MIT 95% figure is, in effect, a census of pilots that skipped step one.
The rollout path, company level. Six steps, in order.
Policy on a page
Which tools, which data, what gets checked by whom. Part VIII’s three-colour version. Buy the business tier the same week.
Inventory
Every use, an owner per row, including the “we use ChatGPT for that” rows. The list is always longer than expected.
One lighthouse process
Top-left of the quadrant. High volume, cheap mistakes, an owner who wants it. Baseline it.
Measure
The scorecard. Shadow, then draft mode. Champions in the team. Weekly readings, blind quality sample.
Scale or stop
Read the card. Scale the process that worked; fix the queue it exposed; stop the one that didn’t. Say so.
Training, by role
Not “AI awareness”. “Here is how this team’s process now runs, and here is what you check.” Two hours, in the tool, on real cases.
Training comes last on purpose. Training people on a tool before the process changed is how “AI awareness” sessions came to exist.
The rollout path at company level is six steps, and they are in this order for reasons. The policy comes first because it takes a week, costs nothing, and removes the most common objection from the people who will have to live with everything else; Part VIII’s three-colour page is enough, and buying the business tier the same week makes most of it self-enforcing. The inventory comes second because you cannot choose a lighthouse from a list you have not made, and because the list always contains surprises. The lighthouse process comes third: one, from the top-left of the value-risk quadrant, with an owner who actively wants it, baselined for four weeks before anything is switched on. Measurement is the fourth step and the previous two slides; shadow mode, then draft mode, champions in the team, weekly readings and a blind quality sample. The fifth step is the decision: scale what worked, fix the bottleneck the pilot exposed, which is usually an approval queue or a data source, stop what did not, and say so publicly so that the next pilot is believed. Training comes sixth and last, by role, as each process scales. Not AI awareness, not prompting workshops, but two hours in the tool on real cases: here is how this team’s process now runs, here is what you check, here is what you never send unread. Training people on a tool before the process has changed is how the AI-awareness industry came to exist, and the KPMG study’s finding that only 47% of employees have had any AI training, while 66% rely on outputs without checking them, is what it has produced.
Change management without the incense.
Resistance is information: people resist tools that add work and threaten their jobs. Those tools do.
Here is change management as mechanism rather than motivation. People resist for five reasons, and each has a cheap answer. Fear that this is the thing that replaces them: say what the freed time is for before the pilot, hire juniors into the redesigned role, and share the scorecard including the bad weeks. Extra work, because the tool is a tab beside the job: put it in the workflow, in the ticket system, the editor, the inbox, and accept that if using it takes more clicks than not using it, nobody will, and they will be right. It does not fit, because a generic tool meets a specific process: redesign the process around the step, which is the MIT report’s “learning gap” seen from the floor, and rely on the team’s champions to find the fit, because the central group cannot. No time to learn, because targets stayed the same and the tool was added: protected hours for champions and a lower target in the pilot month; Microsoft’s 2026 Work Trend Index found only 13% of workers saying their employer rewards reinventing work with AI when the results fall short, and 45% saying it feels safer to focus on current goals. And nobody shares the gains: if a team halves its cost per case and sees none of it, the next pilot gets quietly sabotaged, so decide in advance what the team gets, whether that is headcount held, bonus, or better work. Underneath all five is the shadow-AI reality: KPMG and Melbourne found 57% of employees hiding their AI use, Netskope 47% of workplace use still on personal accounts in 2026. Your people are already doing this; the only question is where. Resistance is usually information: people resist tools that add work and threaten their jobs, because those tools do.
Sources (5)
- https://fbe.unimelb.edu.au/newsroom/media-release-global-study-reveals-trust-of-ai-remains-a-critical-challenge-reflecting-tension-between-benefits-and-risks
- https://www.infotrust.com.au/resource-library/key-findings-netskope-cloud-and-threat-report-2026
- https://finance.yahoo.com/sectors/technology/articles/microsoft-finds-just-13-firms-044659301.html
- https://www.forbes.com/sites/moorinsights/2026/05/19/microsoft-work-trend-index-2026-shows-ai-productivity-is-not-enough/
- https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo
The current board: what a seat costs. Checked October 2026.
| Product | Per user per month | Notes | Data terms |
|---|---|---|---|
| ChatGPT Business (OpenAI) | $20 annual · $25 monthly | 2-seat minimum. Enterprise: custom quote, credit or token billing. | No training on your data by default |
| Claude Team (Anthropic) | $20 annual · $25 monthly | 2–150 seats. Premium seat $100 (5× usage). Enterprise $20/seat + usage at API rates, 20-seat minimum. | No training on your data by default |
| Microsoft 365 Copilot | $30 annual ($31.50 monthly) | Add-on: needs a qualifying Microsoft 365 licence on top. “Copilot Business” for SMB plans listed at $21 (promotional $18). | Enterprise data protection |
| API / tokens (any vendor) | Pay per token used | Cents per routine case; dollars for long agent runs. Unpredictable without a cap. Part III: “cents to dollars per run”. | No training by default; zero-retention options |
For a 200-person company the seats are $40–70k a year. That is the small line. The big lines are the next slide.
The one perishable slide, dated and to be re-checked. As of October 2026 the business-tier chat tools from the two main model vendors cost the same: ChatGPT Business and Claude Team are both $20 per user per month on annual billing and $25 monthly, with a two-seat minimum; Claude Team runs to 150 seats and offers a $100 premium seat with five times the usage, and Claude Enterprise is $20 a seat plus usage billed at API rates with a 20-seat minimum. ChatGPT Enterprise is quote-only with billing by credit or token. Microsoft 365 Copilot is $30 per user per month on an annual commitment, $31.50 monthly, and is an add-on on top of a qualifying Microsoft 365 licence; a cheaper “Copilot Business” for small-business plans is listed at $21 with an $18 promotional price at the time of checking. All business tiers from these vendors exclude your data from training by default, which is the point of buying them, as Part VIII explained. API usage is different in kind: you pay per token, which is cents for a routine triage case and dollars for a long agent run, and it is unpredictable without a cap. For a 200-person company, seats for everyone come to roughly $40,000 to $70,000 a year depending on vendor and mix. That is the small line in the budget. The next slide shows the big ones. Prices are US list; resellers, regions and contracts vary, and these numbers will be wrong within months, so re-check before quoting them.
What it really costs. A 200-person company, year one.
YEAR ONE · 200 PEOPLE · ONE LIGHTHOUSE PROCESS (illustrative; numbers invented) Seats 150 users x $25 x 12 months $45,000 Tokens lighthouse: 6,000 cases/mo x ~$0.08/case x 12 $5,800 Build 1 developer-quarter or vendor set-up + workflow tool $40,000 Data 2 person-weeks cleaning policy docs + CRM fields $6,000 Evals 100-case test set: 3 expert-days; re-run per change $4,000 Review draft mode: 5 min human check x 6,000/mo x $0.60/min $18,000 Owner 10% of one manager, all year $12,000 Champions 6 people x 3 h/week x 48 weeks x $40/h $35,000 Training 200 people x 2 h x $40/h, once the process is live $16,000 TOTAL year one $182,000 of which licences and tokens 28%
Licences are under a third. If your AI budget is a licence line, you have budgeted the easy part and skipped the pilot.
A worked budget for a 200-person company in its first year, with one lighthouse process. Every number is invented, in the style of Part III’s illustrative runs, and the shape is what to take away. The vendor bill, in blue, is seats for the 150 people who will actually use the tool and the tokens for one process at six thousand cases a month: about fifty thousand dollars, which is roughly what the sales deck will show you. The build, in gold, is a developer-quarter or a vendor’s set-up fee plus a workflow tool, two person-weeks cleaning the policy documents and CRM fields the agent will read, and a hundred-case test set built by the people who know the right answers and re-run on every change: around fifty thousand. Neither the data clean-up nor the test set is optional; together they are what distinguishes a pilot from a demo. People’s time, in coral, is the line nobody budgets and everybody pays: five minutes of human review on each draft in draft mode, a tenth of a manager as owner, six champions at three protected hours a week, and two hours of role-specific training for everyone once the process is live. That is around eighty thousand and the largest block. Total about 180,000, with licences and tokens under thirty percent. Three consequences. If your AI budget is a licence line, you have budgeted the easy part. The payback test is simple arithmetic: this lighthouse must clear roughly 180,000 of measured value, about two and a half dollars per case, and only the scorecard can say whether it does. And the human review line is where draft mode quietly eats the saving if the edit rate never falls, which is why Part III tracked it.
Three stories everyone in this room should know.
Cost won. Then quality. Then both.
Feb 2024: its assistant did “the equivalent work of 700 agents”. May 2025: the CEO told Bloomberg cost had been “a too predominant evaluation factor”, quality was lower, and Klarna began hiring humans for support again. Feb 2026: headcount down 49% since 2022; revenue per employee $1.24M. The AI stayed; so did the humans.
A number, a method, and a decade.
Singapore’s DBS reports about S$1 billion of economic value from data and AI in 2025, from 2,000+ models and 430+ use cases, measured against control groups and traced to revenue, cost and risk. It set the S$1bn target in 2022, after years of data work. Self-reported, but with a method you can ask about.
The bot cut jobs. The calls went up.
Jul 2025: Australia’s biggest bank made 45 support roles redundant, citing an AI voice bot. Staff reported call volumes rising and managers pulled onto phones. Aug 2025, after a union dispute: CBA said the roles were not redundant, apologised for the “error”, and offered them back.
Three stories. Klarna is the one everyone half-knows, so get it exactly right. In February 2024 Klarna said its OpenAI-powered assistant handled 2.3 million conversations in its first month and did the equivalent work of 700 full-time agents. In May 2025 its CEO, Sebastian Siemiatkowski, told Bloomberg that cost had been too predominant a factor, that the result was lower quality, and that Klarna was recruiting human support staff again so that there would always be a human if you want one. In February 2026 its fourth-quarter release reported headcount down 49% since the end of 2022 and revenue per employee of $1.24 million. Both halves are true at once: the AI handles most chats and the company is much smaller, and it also had to buy back the human path it had cut. The 700 was an equivalence claim; the headcount fall came mostly from a hiring freeze and attrition. DBS is the quiet positive case. Singapore’s largest bank reports about one billion Singapore dollars of economic value from data analytics and AI in 2025, from more than 2,000 models and 430 use cases, measured against control groups and traced to revenue, cost savings and risk avoidance. It set that target in its 2022 annual report after years of data and platform work. It is self-reported, but it comes with a method you can ask about. The Commonwealth Bank of Australia is the cautionary tale with a happy ending. In July 2025 it made 45 customer-service roles redundant citing an AI voice bot; staff reported call volumes rising and team leaders taking calls; in August, after the Finance Sector Union took it to the Fair Work Commission, the bank said the roles had not been redundant, apologised for the error, and offered the jobs back. The headcount decision had preceded the measurement.
Sources (7)
- https://www.entrepreneur.com/business-news/klarna-ceo-reverses-course-by-hiring-more-humans-not-ai/491396
- https://s205.q4cdn.com/644747736/files/doc_earnings/2025/q4/earnings-result/Q4-25-Klarna-Group-plc-Press-Release.pdf
- https://sayangupta.substack.com/p/crossroads-edition-06-klarna-six
- https://www.dbs.com/newsroom/DBS_named_Asias_Best_Digital_Bank_by_Euromoney_recognised_for_its_AI_leadership_and_responsible_innovation
- https://www.forrester.com/blogs/dbs-banks-billion-dollar-ai-dream-realized
- https://www.computerweekly.com/news/366639844/DBS-rewires-operating-models-for-AI-reasoning-era
- https://www.abc.net.au/news/2025-08-21/cba-backtracks-on-ai-job-cuts-as-chatbot-lifts-call-volumes/105679492
Could you fill in the scorecard?
- Pick one process. What is its cost per case today? Who could tell you by Friday?
- Of the last year’s AI spend, what share was licences, and what share was people’s time? Was the time budgeted?
- Which pilot in your company was allowed to fail? What happened to the one that did?
- If the team halves its cost per case, what do they get?
The first question is the baseline problem in miniature. If nobody can answer by Friday, the AI pilot is not the first project; the measurement is.
Use it Monday
Five things a leader does next week. None of them needs a budget approval. All of them need a name and a date.
Five things, one weekClosing section. Five actions, each of which costs days of someone’s time rather than a line in a budget, and each of which the rest of the deck depends on.
What I would do next week.
Make the inventory.
Every use, including “we use ChatGPT for that”. One row each. What it reads, what it does, which plan. You will find the personal accounts. Buy the business tier the same day.
Pick one process.
Top-left of the quadrant: high volume, cheap and reversible mistakes, someone reads its input all day. Not the most exciting one. The most measurable one.
Name one owner.
The head of that team, not IT, not a committee. They own the baseline, the metric, the test set and the decision at week 12. Give them a champion and protected hours.
Take one baseline.
Four weeks of system-measured time per case, quality sample, rework, cycle time, cost. Before anything is switched on. If it cannot be measured, that is the project.
Write one page of policy.
Which tools, which data, what gets checked by whom. Part VIII’s three colours. Then tell staff what will change, what you will measure, and what you do not know.
None of this is an AI project. All of it is what every AI project that worked had already done.
Five actions for the week after this talk, in the order to do them. Make the inventory first, because everything else depends on knowing what is in use, and because the exercise reliably finds the personal accounts; buy the business tier the same day so the data problem largely solves itself. Pick one process from the top-left of the value-risk quadrant, choosing the most measurable rather than the most exciting. Name one owner, the head of the team whose process it is, and give them a champion and protected hours; they own the baseline, the metric, the test set and the decision at week twelve. Take one baseline over four weeks before anything is switched on; if the process cannot be measured today, the measurement is the project and the AI waits. Write one page of policy in Part VIII’s three colours, then say to staff what will change, what will be measured and what nobody knows. The closing line is the whole deck in a sentence. None of this is an AI project. All of it is what every AI project that produced a number had already done before the model arrived.
The model is the cheap part. Adoption, workflow redesign and measurement are the expensive parts, and they are the parts that decide whether the finance director ever sees it.
Switch on what you pay for. Measure one thing. Then decide.
How to read an AI statistic. Five questions, every time.
Who was asked, and what did they have to gain?
Executives self-reporting EBIT impact (McKinsey) measure belief. Vendors reporting ROI measure marketing. Payroll data (ADP) measures payroll.
Task, person, firm or economy?
A 14% task gain, a 19% hiring gap for 22–25s, a 37% share of firms seeing any EBIT effect and a 1-point shift in occupational mix are all true and all about different things.
Was there a baseline and a control?
Brynjolfsson, Noy & Zhang, Cui and METR had one. The MIT 95% and most “hours saved” surveys did not. “No measurable impact” ≠ “no impact”.
Measured, or recalled?
METR: measured −19%, recalled +20%. Self-reports about AI productivity are wrong in both directions and should be weighted accordingly.
What date, which tools, which country?
Nearly all of it is US, 2023–2026, first-generation tools. The Stanford gap grew from 15% to 19% in a year. Any number older than a year needs re-checking.
Sources used in this deck
Surveys: McKinsey State of AI (Aug 2026); MIT NANDA (Aug 2025); KPMG/Melbourne (2025); Microsoft WTI (2024, 2026); Netskope (2026).
Controlled studies: Brynjolfsson et al. (QJE 2025); Noy & Zhang (Science 2023); Cui et al. (2024); METR (Jul 2025).
Labour data: Stanford Canaries (Aug 2026); Yale Budget Lab (Oct 2025); Challenger (Sep 2026).
Cases: Klarna filings and interviews; DBS newsroom; ABC News on CBA. Full URLs in the notes of each slide.
Every number in this deck passes at least three of the five. Where it passes fewer, the slide says so.
For the technically curious, and for anyone who will be sent a statistic in the next month: five questions to put to every AI number. Who was asked and what did they have to gain: executives self-reporting EBIT impact to McKinsey are measuring their own belief; vendors reporting ROI are producing marketing; payroll records from ADP measure payroll. Task, person, firm or economy: a 14% gain on a support task, a 19% hiring gap for 22 to 25 year-olds in exposed jobs, a 37% share of firms seeing any EBIT effect and a one-point shift in the occupational mix are all true at once and all describe different things; most arguments about AI are people quoting different levels at each other. Was there a baseline and a control group: the four controlled studies had both; the MIT report and nearly every hours-saved survey had neither, and “no measurable impact” means only that nothing was measured. Measured or recalled: METR’s developers measured 19% slower and recalled 20% faster, which should settle how much weight self-reports deserve. And what date, which tools, which country: nearly everything cited here is US data from 2023 to 2026 on first-generation tools, the Stanford gap moved four points in a year, and any number older than a year needs re-checking before it goes on a slide. The panel lists the sources used; the full URLs are in the notes of each slide. Every number in this deck passes at least three of the five questions, and where it passes fewer, the slide says so.
Sources (9)
- https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
- https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo
- https://www.nber.org/papers/w31161
- https://news.mit.edu/2023/study-finds-chatgpt-boosts-worker-productivity-writing-0714
- https://mitsloan.mit.edu/ideas-made-to-matter/how-generative-ai-affects-highly-skilled-workers
- https://getdx.com/blog/metr-study-on-how-ai-affects-developer-productivity/
- https://digitaleconomy.stanford.edu/news/canariesaug26/
- https://budgetlab.yale.edu/research/evaluating-impact-ai-labor-market-current-state-affairs
- https://www.challengergray.com/wp-content/uploads/2026/09/Challenger-Report-August-2026.pdf