Monitoring AI Systems: The Failures That Do Not Crash

Most monitoring set-ups were built for software that fails loudly. A server crashes, a queue backs up, an error rate spikes, an alert fires, someone gets paged. That model works because traditional software fails by breaking. An AI system rarely breaks. It keeps responding, in fluent, well-formatted, entirely plausible sentences, while being wrong.

This is the property that catches most teams out once an AI feature moves from pilot to production. The system does not throw an exception when it misreads a contract clause or miscategorises a support ticket. It returns an answer with the same confidence whether that answer is right or wrong, and every dashboard built to watch for crashes stays green throughout. Uptime tells you a system is running. It tells you nothing about whether it is correct.

Why does a healthy dashboard not mean a healthy AI system?

Conventional application monitoring answers a narrow question: is the system available and responding within an acceptable time? For a database or a web server, availability and correctness are close cousins. If the query ran and returned a result in the expected shape, the result is very likely right. An AI system breaks that assumption: it can be fully available, well within its latency budget, and wrong on a material share of requests, with none of the standard signals showing it.

The failure mode that matters here is not the model going down. It is the model staying up and drifting away from correct, one plausible-sounding answer at a time. That is a different problem from anything most engineering teams have instrumented for, and it needs its own layer of observability sitting alongside, not instead of, the infrastructure monitoring already in place. We think about this as part of the same progression we set out in the AI transformation ladder: as AI moves from an assistive tool to something making or shaping decisions unsupervised, the monitoring has to grow up alongside it.

What should you instrument on every request?

Three things, at minimum, on every call to the model, not sampled after the fact:

• Cost. Token usage per request, aggregated by feature, by user segment, and by time of day. A sudden jump in tokens-per-request on an unchanged prompt is often the first visible symptom of something going wrong upstream.
• Latency. Not just the average, the full distribution, including the tail. A system that is fast on the 50th percentile and unacceptably slow on the 99th percentile will look fine on a dashboard that only shows the mean.
• A quality signal. This is the one that is missing almost everywhere, and it is the one that actually tells you whether the system is doing its job. Cost and latency describe the plumbing. Only a quality signal describes the water coming out of the tap.

A quality signal does not need to be exotic. Two practical versions cover most cases. The first is sampled grading: take a percentage of live outputs, score them against a rubric (either by a person or by a second model checking the first, with a person auditing that check periodically), and track the score over time by feature and by version. The second is a downstream correction rate: how often does a human change, override or reject what the system produced before it takes effect? A rising correction rate on an unchanged workflow is one of the most reliable early-warning signals available.

The instrumentation only earns its keep if someone looks at it on a schedule, and all three belong on one dashboard, because a change in one is often the first clue to a change in another.

What does the timing of a failure tell you?

When something is wrong, when the system starts to look wrong, is itself diagnostic information. Four patterns show up repeatedly, and each points somewhere different.

Wrong from the very first response. If a system has never worked correctly, not on day one, not on the first test case, the fault is almost always in the specification: the prompt, the instructions, the definition of the task itself. The fix is to check what the system was actually told to do against what it was meant to do, rather than tuning around the symptom.

Degrades as a single session grows. A system that starts well within a conversation and gets worse as that conversation lengthens is usually running into context limits: the earlier instructions, constraints or facts are being pushed out or diluted as more gets added. The fix here is architectural, not cosmetic. It means re-injecting the constraints that matter, summarising rather than accumulating history, or capping how long a session runs before reset.

The same specific error, repeating. One recognisable mistake showing up again and again, rather than a spread of different errors, usually means a wrong tool or a wrong model choice for that particular sub-task. The system is being asked to do something it is structurally not well suited to. The fix is to change what is doing the work for that step, not to keep rewording the instructions to it.

Worked last month, no code change. This is the pattern that unsettles teams most, because nothing on their side moved. The explanation is decay somewhere it isn't easy to see: a change in the data the system is fed, a configuration drift nobody flagged as significant, or a silent update to an upstream model provider's own service. None of these show up in your own commit history. One common version, a document store outgrowing its index, is covered in why your AI assistant gives out-of-date answers. The only defence is comparing current behaviour against a fixed baseline on a schedule, not relying on someone noticing.

Why is slow-burn degradation the hardest case?

The four patterns above are all, eventually, noticeable. Slow-burn degradation is different, and it is the hardest failure mode to catch precisely because the symptoms are visible the entire time, just never in a form that gets connected to a cause.

Quality slides gradually. A support-triage system starts routing a slightly higher share of tickets to the wrong queue. A drafting tool starts producing content that needs slightly more editing. Each individual instance looks like ordinary variance and gets waved off. No single incident is bad enough to trigger an investigation. It is only when someone pulls a quarter's worth of data together, usually at a quarterly review, that the trend becomes visible and someone finally asks how long this has been happening.

The only real defence against this is exactly the sampled quality tracking described above, held over a long enough window that a slow trend has somewhere to show up. A weekly or monthly view of the correction rate and the grading score, plotted over time, turns a slow burn into something you can see arriving.

Why can't the model be its own audit log?

When something does go wrong and needs to be reconstructed afterwards, the record of what the system actually did has to come from outside the model. Asking a language model to report on its own actions after the fact is not an audit trail. It is the model generating another plausible-sounding piece of text, exactly as prone to confident error as anything else it produces.

A usable audit log is built deterministically, in the application layer that sits around the model: every action taken, every external call made, every record changed, written down by ordinary application code at the point the action happens, independent of what the model would say if asked about it. This matters for debugging, and it matters more for anyone answering to a regulator, an auditor or a client about how an automated decision was reached. We cover the wider governance obligations this feeds into in our AI governance framework for UK businesses, but the underlying engineering point stands on its own: if the log depends on the model being honest and accurate about itself, it is not a log, it is a second output you also have to check.

Is sampled human review a one-off launch task?

Most teams do some version of human review before a system goes live: a batch of test cases, a sign-off, a go-ahead. That review then stops, on the reasonable-sounding assumption that a system checked once is checked. It isn't. The model behind the system can change without your knowledge, the data feeding it can shift, and the mix of real-world inputs it sees in production is never quite the same as the test set used before launch (building that test set properly is its own discipline: how to test an AI agent before you trust it).

Sampled human review belongs on the same footing as the automated metrics: an ongoing, scheduled instrument, not a launch-week activity that gets ticked off and forgotten. A fixed percentage of live outputs, reviewed on a fixed cadence, by someone with the authority to act on what they find, does more to catch real degradation than any amount of additional automated alerting on its own. The two work together: automated signals tell you where to look, and human review tells you whether what you are looking at is actually a problem.

Building this properly, cost and latency tracking, a genuine quality signal, deterministic audit logging and a standing human review cadence, is engineering work, not a bundled checkbox. It is the kind of work we build into every AI system we deliver through our AI implementation service, because a system nobody is watching for correctness is a liability wearing the shape of a feature. If you want a clear-eyed view of where your own AI monitoring stands today, our diagnostic review is a straightforward place to start.

Frequently asked questions

What is the single most useful metric to add if we only have uptime and latency today?

Add a downstream correction rate first: track how often a human changes, overrides or rejects the system's output before it takes effect, broken down by feature and tracked over time. It requires no new grading infrastructure, it uses a signal your team is probably already generating as part of normal work, and a rising trend is one of the clearest early warnings of quality problems available.

How much of our output do we actually need to sample for quality review?

There is no universal percentage, because it depends on volume and on how much damage a bad output can do. A high-stakes, low-volume workflow, such as anything touching a contractual commitment, can justify reviewing close to all of it. A high-volume, low-stakes workflow can work from a smaller statistical sample, provided the sample is drawn consistently and reviewed on a fixed schedule rather than opportunistically.

Can we rely on the model itself to flag when it is uncertain or has made a mistake?

Treat any self-reported confidence or self-assessment from the model as a data point worth logging, not as ground truth. A model can express high confidence in a wrong answer and genuine uncertainty about a correct one, because that expression is generated the same way as the rest of its output. It is useful as an additional signal to correlate against your independent quality metrics, never as a substitute for them.

Does this level of monitoring only matter for large-scale AI deployments?

No. The size of the deployment changes how much monitoring infrastructure is worth building, not whether the underlying risk exists. A single AI-driven workflow making a handful of decisions a day can still produce a confidently wrong answer on the first request, and the cost of that one wrong answer does not scale down just because the volume is low.

John Kelleher is a Claude Certified Architect (Foundations and Professional) and leads SpotDev, a Claude Registered Partner and OpenAI Select Partner.