Why AI Pilots Fail in Production

Most AI pilots we see work. That is not the problem. A demo running against ten curated examples, or a proof of concept a project team has been quietly using for a few weeks, will almost always show a green tick. The problem is that the conditions which make a pilot look good are precisely the conditions production removes.

Development hides the failures production reveals. Pilots run on small, clean, hand-picked inputs with a forgiving audience and no real cost pressure, so none of the things that break a system in the wild ever get a chance to show up. If a failure mode only appears at volume, with messy inputs, under cost constraints, it was never actually tested, no matter how many times the pilot ran.

Why does a working pilot tell you so little about production?

Scale is the first shock. A pilot handling twenty requests a day from one enthusiastic team looks nothing like the same workflow rolled out across a department or wired into a customer-facing process. Production volumes typically run many times higher than whatever the pilot touched, and in our experience the jump is rarely gradual. That is not a linear stretch of the same conditions, it is a different regime: rate limits that were never hit start firing, background jobs queue up, and a system that quietly did its job for a project sponsor now has to do it reliably for everyone else.

The second shock is the input itself. Pilots are usually fed short, tidy examples because someone built them to demonstrate the idea, not to break it. Real inputs are longer, messier, and less predictable: a support ticket with three unrelated issues bolted together, a document with tables and scanned pages mixed in, a customer message that trails off halfway through a sentence. Systems built and tested on tidy fixtures hit context and cost ceilings the pilot never came close to, because the pilot never had to process the input that real customers or real colleagues actually produce. We wrote about this same gap in the context of turning a working chatbot demo into a production tool: see our piece on taking a ChatGPT prototype into production, which covers the same pattern from a slightly different angle.

Where does the cost model break down?

Teams that model cost at all usually model it on the average request, and that is the mistake. Token and cost distributions in production are skewed, not even. A small tail of unusually long requests, whether that is a huge document, a long conversation history, or a request that triggers several tool calls in sequence, drives a disproportionate share of total spend. Average-case costing looks fine right up until the month a handful of outlier requests account for a disproportionate share of the bill, and nobody budgeted for that because nobody looked at the tail.

The fix is not complicated, but it has to happen before go-live rather than after the first invoice. Take a sample of genuinely production-shaped inputs, including the longest and messiest ones you can find, and cost the system against that distribution rather than the mean. If the answer is uncomfortable, that is useful information gathered on your own terms, not the vendor's invoice.

What happens when nothing goes wrong until it does?

The most common gap we find in a pilot moving to production is not a missing feature, it is a missing error path. A pilot has no plan for timeouts, no plan for rate limits, no plan for a tool call that comes back with a malformed or unexpected response, because none of those things happened often enough during testing to force a decision. In production, at ten or a hundred times the volume, they happen constantly.

The instinct once a failure does show up is to retry. That instinct is only half right. Retrying everything blindly duplicates side effects: an email sent twice, a record created twice, a payment or a case logged twice, because the first attempt actually succeeded and the system retried anyway on a network blip. The distinction that matters is between retriable failures (a timeout, a rate limit, a transient network error, where trying again is safe and often correct) and terminal failures (a malformed request, an authorisation error, a response that will never succeed no matter how many times you send it, where retrying just wastes time and money while the underlying problem sits unresolved). A production system needs to know which is which for every call it makes, and a pilot almost never does, because it was never asked to.

Why pin a model version if newer is meant to be better?

Pilots are usually built and run against whatever model version happens to be live at the time, and that is fine while a handful of people are testing it. It stops being fine once a workflow depends on consistent behaviour. Model providers update and retire versions on their own schedule, and a silent upstream change, a shift in tone, a different way of formatting an answer, a change in how a model handles an edge case, can move a production system's output in ways nobody signed off on.

Pinning the model version you deploy against turns that risk into something you control. An upgrade becomes a gated release: test the new version against the same production-shaped inputs, compare the outputs, and decide deliberately when to move, rather than discovering the change because a customer complained. This is one of the clearer markers of where an organisation actually sits on the path from experimenting with AI to running it as infrastructure, something we set out in more detail in our piece on the AI transformation ladder.

Is the same failure happening inside your own organisation?

There is an organisational version of this exact pattern, and it is easy to miss because nobody built it on purpose. Someone in a team puts together a helpful internal tool: a script that summarises reports, a small assistant that drafts responses, a workflow that pulls data together automatically. It works, people start relying on it, and other teams start relying on the people who rely on it. Nobody assigned an owner. Nobody set up monitoring. Nobody wrote an error path, because it was never meant to be more than a personal convenience.

Months later that tool is load-bearing. It sits underneath a process other teams now depend on, with no one accountable when it breaks, no alert when it silently starts returning wrong answers, and no plan for what happens when the person who built it moves to a different project or leaves. This is the same production gap as the technical one above, just discovered organisationally rather than architecturally: something that was never tested against real dependency is now depended on.

What does a proper go-live gate look like?

The discipline that closes this gap is not complicated, but it does have to be deliberate. Test with production-shaped data before launch, not the tidy fixtures used to build the thing; how to test an AI agent before you trust it sets out that discipline in full. Model cost at production volume, against the real distribution of request sizes, before committing budget or telling anyone the numbers. Treat go-live as a gate with defined evidence: a stated error rate under load, a cost-per-request figure that has actually been checked against the tail rather than the average, a documented answer for what happens on a timeout and what happens on a terminal failure, and a named owner once it is live. None of that requires exotic engineering. It requires treating the move from pilot to production as a distinct piece of work with its own deliverables, rather than a formality once the demo has gone well. Where we get involved is usually at exactly that point: a pilot that works, and a client who wants a straight answer on what it will take, and cost, to run it properly. Our AI implementation service is built around that gap specifically, and if you are not sure whether what you have built is production-ready or just a demo that has not failed yet, our diagnostics assessment is a reasonable place to start finding out.

Frequently asked questions

How do I know if my AI pilot is actually ready for production, or just working well as a demo?

Test it against a sample of real production-shaped inputs, not the examples it was built and demonstrated on, and check three things: does it stay within cost and context limits on the longest and messiest inputs you can find, does it have a defined behaviour for timeouts, rate limits and bad tool responses rather than silence or a blind retry, and is there a named owner and some form of monitoring once it goes live. If any of those three is missing, it is a demo, however well it has performed so far.

What is the difference between a retriable failure and a terminal failure in an AI system?

A retriable failure is transient: a timeout, a rate limit, a dropped connection, where the original request likely did not succeed and trying again is safe and usually correct. A terminal failure is one that will not succeed no matter how many times you send it: a malformed request, an authorisation problem, an input the system cannot handle. Retrying a terminal failure wastes time and money without fixing anything, and retrying indiscriminately risks duplicating side effects such as sending the same email or creating the same record twice when the first attempt actually went through.

Why does production cost so much more than the pilot suggested?

Because pilot costing is usually based on an average request, and real usage is skewed: a small proportion of unusually long or complex requests account for a disproportionate share of total spend. If you only cost the typical case, you have not costed the system, you have costed the easy part of it. Modelling cost against a genuine production distribution, including the outliers, before committing budget is the only way to avoid an unpleasant surprise on the first full invoice.

What should we do about internal AI tools that were never meant to become critical infrastructure?

Audit what is actually being relied on rather than what was formally sanctioned. Once you find a tool that other teams now depend on, treat it exactly as you would any other production system: assign an owner, add basic monitoring so failures are visible rather than silent, and define what happens when it breaks or returns a wrong answer. The technical fix is often small. The harder part is simply noticing that an informal convenience has quietly become something people can no longer do their jobs without.

John Kelleher is a Claude Certified Architect (Foundations and Professional) and leads SpotDev, a Claude Registered Partner and OpenAI Select Partner.