How to Test an AI Agent Before You Trust It

Most AI agent projects get judged on a demo. Someone runs three or four well-chosen questions through the system, the answers come back fluent and confident, and the room decides it's ready. That's the wrong test, and it's the test almost everyone runs.

A demo shows you how the system behaves when you already know what a good answer looks like and you've picked questions it's likely to handle well. It tells you almost nothing about how it behaves on the query you didn't think of, the record with a gap in it, or the user who phrases the same request oddly. An AI agent earns production trust by passing an evaluation you built before you wrote the first prompt, not by performing well while you were watching.

What does "success" actually mean here?

Before you optimise anything, write down what a correct answer looks like. Not in general terms: specifically, for this task. What counts as an acceptable refusal versus an unhelpful one? What format is non-negotiable? Is a slightly wrong number worse than a slightly wrong word? Usually yes, but neither triggers an obvious error, which is exactly why the distinction has to be written down rather than judged case by case. Without this written down, every review of the agent's output turns into an argument about vibes, and "looks about right" wins arguments it has no business winning. This is the step people skip because it feels like paperwork. It's the step that determines whether everything after it means anything.

Why does a handful of good examples prove nothing?

Watching an agent handle a few cases you picked yourself is watching a highlight reel, and highlight reels exist to flatter. The fix is a proper test set: a modest number of real, varied cases pulled from actual data or actual past enquiries, not invented for the occasion. It needs boundary cases (right at the edge of what the agent is supposed to handle), adversarial cases (malformed input, contradictory instructions, attempts to get it to say something it shouldn't), and cases where the correct answer is "I don't know" or "no". A handful of varied real cases beats three hand-picked good ones by a wide margin, because the three good ones were never going to fail. That's rather the point of picking them.

Why do exact-match tests mislead you on a system like this?

If you come from conventional software testing, the instinct is to check the output string against an expected string. That instinct is wrong for a language model, because the same correct answer can be phrased a dozen different ways, and none of them will match a fixed string. Run exact-match tests against a non-deterministic system and you get false failures on good answers and, worse, false passes when a broken answer happens to contain the right substring. The test has to judge whether the content is right, not whether the wording matches a script. That distinction is why evaluating an agent is a different discipline from unit-testing a function, even though the instinct to reach for the same tools is understandable.

How should the checks be graded, cheapest first?

Not every check needs the same amount of scrutiny, and treating them all the same wastes money and slows everything down. A practical approach is to grade in three tiers, cheapest first:

• Deterministic code checks. Is the output valid JSON, does it have the required fields, are the numbers in a sensible range, do the links actually resolve. This catches a surprising amount for almost no cost, and it should run on every single output before anything more expensive gets involved.
• Model grading against a written rubric. For the semantic questions, like whether the answer is actually correct and whether it addresses what was asked, a second model can grade the output against a rubric you've written. The rubric has to be explicit. Ask a grading model to judge "quality" with no criteria and it invents its own standard, which defeats the purpose.
• Human review. Reserved for genuine judgement calls, for the cases the automated graders disagree on, and for periodically sampling the automated grading itself to check it hasn't drifted. This is the most expensive tier, so it should be the smallest by volume, not the default.

What specific failure types should the test set go looking for?

Generic testing finds generic problems. The failures that matter with language models are specific, and the dataset has to be built to provoke them deliberately rather than hope they surface on their own:

• Plausible-but-unsupported claims. An answer that reads as authoritative but isn't actually backed by anything in the source material it was given.
• Fabricated specifics. An invented name, figure, date or citation dropped into an otherwise sensible-sounding response.
• Confident tone masking uncertainty. A guess stated as fact, with none of the hedging the situation actually calls for.
• Internal contradictions. The agent states one thing early in a response, or early in a conversation, and contradicts it later without noticing.
• Format drift. The output holds its required structure for the first few turns or the easy cases, then quietly breaks it under pressure.

It's worth revisiting what actually turns a piece of software into an AI agent that takes real actions rather than just answering questions, because the more consequential the actions it can take, the more expensive each of these failure types becomes if it reaches production undetected.

What happens when the prompt, the model or the data changes?

The evaluation isn't a one-off hurdle you clear before launch. It's a gate you run again every time something underneath the agent changes: a new prompt, a different model version, a new data source connected to it. "We upgraded the model" should never be an assumption that quality held steady. It should be a trigger that runs the full test set again before anyone ships the change. Teams that skip this find out about a regression from a client, weeks after it started, rather than from their own tooling on the day it happened. The evaluation gate is one piece of the larger move from pilot to production; why AI pilots fail in production covers the rest. This is also why the choice between a custom agent and a standard tool matters for testing, not just for cost: a custom-built agent inherits whatever evaluation discipline you put around it, while an off-the-shelf tool gives you far less visibility into what changed on the vendor's side.

This is the standard we insist on before any agent of ours goes near production data or a live customer, and it's a core part of how we scope AI implementation work. If you're not sure whether something already running in your business would survive this kind of scrutiny, that's precisely the sort of question a structured diagnostic review is built to answer before it becomes a live incident instead of a test failure.

Frequently asked questions

How many test cases do we need before an agent goes live?

There's no universal number, because it depends on how varied the real-world inputs are, but a common failure is stopping at three or four cases because they all pass. A more useful benchmark is coverage: you want enough cases, typically dozens rather than a handful, to include every boundary condition, every adversarial pattern you can think of, and every "this should be refused" scenario the agent will actually meet in production, not just the cases that make it look good.

Can a language model grade its own output fairly?

A second model can grade another model's output reasonably well, but only against a rubric that's written down in advance and only when you periodically check its grading against human judgement. Left ungoverned, a grading model tends to reward fluent, confident-sounding answers over correct ones, which is exactly the bias the evaluation exists to catch in the first place.

How often should the evaluation be re-run once the agent is in production?

Every time the prompt, the underlying model or the connected data changes, and on a regular schedule even when nothing has visibly changed, since some vendor-side model updates happen without an obvious announcement. Treat it as a regression gate that sits between any change and deployment, not as a one-off certificate you earned at launch.

Is this different from testing a normal software feature?

Yes, in one important respect: normal software is deterministic, so a fixed test with an expected output works. A language model can phrase a correct answer many different ways and phrase an incorrect one convincingly, so the test has to judge meaning and support for claims, not string matching. The engineering discipline is the same in spirit, building repeatable checks before you trust a system, but the checks themselves have to be built differently.

John Kelleher is a Claude Certified Architect (Foundations and Professional) and leads SpotDev, a Claude Registered Partner and OpenAI Select Partner.