Insights / Reliability
The pilot worked, and that is not evidence
A pilot runs under conditions production never repeats. Before a good result becomes a decision to scale, four things have to be tested that a pilot is structurally unable to test.
A successful pilot is the most persuasive artifact in enterprise technology and one of the least predictive. It demonstrates that a thing can work. The question being asked of it is whether the thing will keep working, at volume, without its author present, and a pilot is structurally unable to answer that.
This is not an argument against piloting. It is an argument for knowing what a pilot measured, so that the decision made on the strength of it is the decision the evidence supports.
What a pilot quietly holds constant
Four conditions are almost always true during a pilot and almost never true afterwards. Each one inflates the result.
The inputs were curated. Pilot data gets selected, often unconsciously, from cases someone already understood. The malformed records, the entries with a decade-old convention, the ones where a field means something different because a since-departed employee used it for notes, are underrepresented. Production has no such filter.
The operator was motivated. The person running the pilot wanted it to succeed. They corrected inputs without noticing they were correcting them, retried the odd failure, and absorbed friction that a routine user would escalate or work around. Their goodwill is not part of the system, and it does not transfer.
The author was watching. During a pilot, whoever built the thing is present and attentive. Failures get diagnosed in minutes. That responsiveness is a property of the pilot period, not of the system, and it disappears exactly when the volume rises.
Nothing upstream changed. A pilot is short. The systems it depends on kept their schemas, their permissions, and their response formats for its entire duration. That is a statement about six weeks, not about the API.
A pilot measures capability. Scaling requires evidence of durability. These are different claims supported by different tests, and treating the first as though it establishes the second is the most common way a promising project turns into an expensive one.
Four tests worth running before scaling
Run it on the ugly data. Deliberately assemble the cases nobody chose: the oldest records, the ones with missing fields, the ones that were migrated from a previous system. The failure rate on that set is much closer to the production rate than the pilot number is. If it is unacceptable, that is a finding worth having before commitment rather than after.
Hand it to somebody who does not care whether it works. A user with no stake will reveal in a day what a motivated operator smooths over for a month. Where do they get stuck, what do they misread, and what do they do when it fails? That last question matters most, because the answer defines the exception path the design has to account for.
Leave it alone. Run it for a period with no intervention and no monitoring by its author, and record what happens. A system that only works while it is being watched has not been shown to work. This test costs nothing but patience and it is the one most often skipped.
Break a dependency on purpose. Change a field name, revoke a credential, take an upstream service away. What matters is not whether the system survives, which it usually should not, but whether it fails loudly. Silent failure at pilot scale is an inconvenience. Silent failure at production scale is weeks of wrong output that nobody questioned.
The volume question is separate
Even a system that passes those four tests faces one more change that a pilot cannot simulate: the exception path stops being manageable. At pilot volume, a ten percent exception rate is a handful of cases somebody handles between other work. At production volume it is a queue, and a queue needs an owner, a priority order, and somewhere to escalate.
Most automations that are described as having failed did not stop working. They kept working on the cases they covered, and the residue of cases they did not cover accumulated until it became somebody's whole job. The system was fine. The design had simply never accounted for what happens to the remainder, because at pilot scale the remainder was invisible.
So the useful question after a successful pilot is not whether to scale it. It is what the exception rate multiplied by production volume actually produces, who is going to own that number, and what the plan is for driving it down. A pilot that comes with an honest answer to that is worth far more than one with a better headline result.