Insights / Reliability
Why AI pilots die in month three
The failure is rarely the model. It is that nobody instrumented the thing, so the first upstream change breaks it silently and the outage is measured in weeks.
There is a specific pattern to how automation projects fail, and it is consistent enough to predict. The build goes well. The demo is genuinely impressive. It runs correctly for a few weeks. Then a vendor rotates a credential, or a form gains a field, or an API deprecates a version, and the automation stops. Nobody is told. The work quietly reverts to being done by hand, and it takes somewhere between three weeks and three months for anyone to notice that the system they paid for is no longer doing anything.
By the time it surfaces, the conclusion has already formed: the AI project did not work. In most cases that is the wrong diagnosis. The build worked. What was missing was every operational practice that keeps software alive after the person who wrote it stops looking at it daily.
Silent failure is the actual problem
A web server that goes down announces itself. Customers cannot reach the site, someone calls within minutes, and the incident begins. Automation fails differently. When a nightly sync stops running, nothing visibly breaks. The records it would have created simply do not exist, and absence is much harder to notice than error.
This is why the standard infrastructure question is not is it working but how would I find out if it were not. Those are different questions and only the second one produces a monitoring design. An automation that processes fifty records a day should alert when it processes zero, and it should alert when it processes five hundred, because both are evidence that something upstream changed.
The practical test. Ask of any automation already running: if this stopped working tonight, what would tell you, and how long would it take? If the answer is that somebody would eventually notice the work piling up, it is not monitored. It is unattended.
The four things that actually break
Across integration work, failures concentrate in a small number of places. None of them involve the model doing something surprising.
1. Authentication expires
This is the most common single cause, and it is almost always avoidable. OAuth tokens expire, refresh tokens get revoked, service accounts get disabled during staff changes, and API keys get rotated by a security policy nobody connected to the automation. The build works during testing because the credential is fresh.
There is a particularly expensive version of this involving Google integrations. An application left in testing mode rather than properly published issues refresh tokens that expire after seven days. The automation demos perfectly, runs for a week, and dies on day eight. The fix is architectural and has to be decided before the build, not after: the application lives inside the client's own workspace as an internal app, which removes the verification requirement entirely and keeps the credentials under the client's control.
2. Upstream schemas drift
Someone adds a required field to a form. A CRM administrator renames a pipeline stage. A finance lead adds a column to the export. Each is a reasonable change made by someone with no reason to know an automation depends on the old shape. Code that assumes a fixed structure fails on the first one of these, and code that silently skips records it cannot parse fails worse, because it keeps reporting success while dropping data.
3. Volume moves outside the tested range
Automations are usually built and tested against ordinary volume. Then a busy period arrives, rate limits are hit, retries stack up, and jobs begin timing out. The system does not fail cleanly; it fails partially, which is considerably harder to detect and to recover from.
4. The exception nobody mentioned
This one traces back to scoping rather than engineering. Every real process has cases that are handled differently, and they are frequently omitted from the description of how the work is done, because to the person doing it the exception is obvious rather than notable. The automation meets it in week six and has no branch for it.
What production practice looks like applied here
None of this is novel. It is standard operational discipline from infrastructure work, applied to a category of software that has largely been built without it. Four practices carry most of the weight.
Alert on absence, not just errors. Heartbeat monitoring on scheduled jobs catches the case where a process stops running entirely, which error-based alerting by definition cannot. A job that fails loudly is a good outcome. A job that stops existing is the dangerous one.
Alerts must reach a person. A dashboard nobody opens is not monitoring. An alert that routes to a shared inbox with four hundred unread messages is not monitoring either. The alert needs a destination that interrupts someone, and a documented expectation of who responds.
Fail loudly and stop. When an automation encounters something it does not understand, the correct behaviour is almost always to halt and escalate, not to continue with a default. Systems that guess produce data corruption that surfaces months later and is expensive to unwind.
Write the runbook for someone else. Documentation written by the builder, for the builder, is a set of reminders. Documentation that is actually useful assumes the reader has not seen the system before, and covers what it does, what it touches, how to tell whether it is healthy, and what to do when the alert fires at an inconvenient hour.
Why this matters commercially
The reason the instrumentation question is worth this much attention is that it determines whether the investment survives. An automation removing five hours a week returns roughly two hundred and sixty hours a year, and that figure is only real if the thing is still running in month eleven. A build that silently stops in month three delivered a fraction of what it was scoped to deliver, and it also produced something more damaging: an organisation that now believes this category of work does not deliver.
Instrumentation is a small fraction of build effort. It is the difference between a system that fails visibly and gets fixed in an afternoon, and one that fails quietly and gets written off.