How to measure whether an automation actually saved anything

Most reported automation savings cannot withstand a direct question. The number was produced after the fact, by asking people how much time they think they used to spend, and it is almost always wrong in the same direction. Recalled effort inflates, particularly for work that was annoying.

This matters for a practical reason beyond honesty. An organisation that cannot verify a claimed saving will not fund the next project on the strength of it, and an inflated number that gets tested later damages credibility more than a modest number that holds.

The baseline has to exist before the build

The single decision that determines whether measurement is possible is when the baseline gets recorded. After delivery, the pre-automation state is only available through recollection, and the people recalling it have watched a system replace their work, which does not encourage conservative estimates.

A usable baseline needs four values, captured before anything changes:

• Frequency. How many times per month the task runs. Counted, not estimated.
• Duration. Minutes per instance, ideally timed across several real instances rather than asked about.
• People. How many individuals perform it, since work distributed across a team is routinely undercounted when only one person is asked.
• Rework. How often the task has to be redone or corrected, and what that costs when it happens.

Measuring the same thing afterwards

The post-delivery measurement has to be the same measurement, which sounds obvious and is routinely violated. Three specific errors account for most overstatement.

Counting the whole task when part of it remains. If an automation handles preparation and a person still reviews and approves, the saving is the preparation time, not the full task. Reviewing output is faster than doing the work, but it is not free, and the review step is a real cost that belongs in the calculation.

Ignoring the exception path. An automation covering eighty-five percent of cases leaves fifteen percent to be handled manually. Those cases are usually the slower ones, so the residual time is a larger share of the original than the case count suggests. Measuring only the happy path produces a number that will not survive a month of real operation.

Omitting the maintenance the system introduced. Automations require attention: alerts get investigated, credentials get rotated, upstream changes get accommodated. It is typically small, but it is not zero, and a saving reported gross of it is not a net saving.

A worked example

A billing process performed by five people, each spending roughly six to seven hours a week on it, across several sub-tasks: authorisation checks, split-payment setup and maintenance, cancellations, renewals, and credit calculations. Per person that is about twenty-five to thirty-two hours a month. Across five people who each carried the volume independently, roughly thirty hours a week team-wide.

Note the structure of that statement. The per-person figure comes first and is derived from named sub-tasks. The team figure is the product of two numbers a reader can check. If someone disputes the total, the disagreement resolves against a specific sub-task rather than against a headline number with nothing underneath it.

Note also what it does not claim. It does not say the work vanished, and it does not include the time now spent monitoring the automation and handling the cases it escalates. Those are subtracted before the figure is described as a saving.

When the honest answer is that it did not work

Occasionally the measurement shows the saving is much smaller than projected, or absent. The usual causes are that the exception rate was higher than scoping suggested, that the review step consumed most of the gain, or that the task was less frequent than believed.

That result should be reported as plainly as a good one. It is the only way the measurement means anything the next time, and the cause is almost always informative: a higher exception rate than expected is a finding about the process, not merely a disappointing outcome. If the hours did not move, the engagement did not achieve what it was scoped to achieve, and saying so is what makes the numbers that do hold up worth believing.