Did it actually work? How to measure an AI project honestly

At renewal time everyone has a story — the supplier's is glowing, the loudest incident says otherwise. Neither is a measurement. Here are the three numbers that are.

Six months in, somebody asks whether the AI project was worth it, and the room fills with anecdotes. The supplier has a slide saying yes. The person who caught the one embarrassing mistake in March says no, with feeling. Both are stories. A project that cost real money deserves an answer made of numbers — and the good news is there are only three of them, plus one thing you must resist.

No baseline, no verdict

The whole exercise rests on knowing what the work cost before. If a fortnight of counting happened at the start — how many items a day, minutes per item, how often it went wrong — you're set. If it didn't, reconstruct one now, honestly: pick the nearest team still doing the job the old way, or dig out last year's throughput from whatever system recorded it. An imperfect baseline written down beats a confident memory every time, because memory reliably improves the old days. (This is why our workflow review starts with two weeks of counting, and why we tell clients the count matters more than the report around it.)

The three numbers

Hours returned. Not "the team feels less busy" — count it. The task took the team eleven hours a week in March; the humans now spend ninety minutes reviewing what the agent did. That difference, times fifty-two, is the headline. Count the review time honestly: supervising an agent is work, and pretending otherwise flatters the result.

Errors, both directions. The rate at which things went wrong before — misfiled, mistyped, missed entirely — against the rate now. Two honesty rules. Your old error rate was higher than anyone believed, because nobody was logging it; the agent's errors, by contrast, are all logged, which makes them look worse in comparison. And the errors change species: fewer misfiles, but the occasional confident mistake a human wouldn't have made. Count both kinds, and note what each costs to put right.

Turnaround. The time from a thing arriving to a thing done — the number your customers experience. A document that used to sit in an inbox until Thursday and now gets acknowledged and filed in four minutes changes how your business feels to deal with, even if hours-saved is modest. This one most often justifies a project on its own.

The vanity metrics you'll be shown instead

When those three numbers are missing, others rush in wearing suits. "Documents processed: 14,000" is volume with no comparison — the old process also processed everything, eventually; the question was never whether but at what cost. "99% accuracy" demands two follow-ups: measured on what, and by whom? An accuracy figure without a denominator and an independent check is a mood. "Time saved: £38,000" is usually hours multiplied by a salary assumption nobody agreed to — ask for the hours and do your own multiplication. And usage statistics — logins, queries, adoption — measure enthusiasm, not results. None of these are lies, exactly. They're answers to easier questions.

When to measure

Not in week two. The first month flatters everyone: the team is paying extra attention, the supplier is watching closely, and the workload happens to be whatever the workload happened to be. Take your readings at month three and again at month six, when the novelty has worn off, the reviewer has settled into spot-checks, and the system has met a holiday, a busy period and at least one bad day. The month-six number is the true one.

What to do with the answer

Three outcomes. It clearly worked: the numbers beat the baseline by more than the system costs to run and mind. The move is the second workflow, which inherits the plumbing you already paid for and should be judged the same way. It clearly didn't: switch it off. This is a better result than it feels like — you've learned a true thing for the price of one build, you keep the connections and logging for a better-chosen project, and a system that can be switched off cleanly is evidence it was built properly in the first place. The murky middle — some gain, not obviously enough — is nearly always a scope problem rather than a technology problem: the agent was pointed at a job that was 70% rule-shaped, and the 30% is eating the savings. Narrow the scope to the part it does well, hand the rest back to people, and measure again.

One last thing, and it's the cheapest insurance in this whole field: agree the three numbers, the baseline and the review dates before the project starts, in writing. A supplier who welcomes being measured is telling you something. So is one who doesn't.