Why Your AI Assistant Gives Out-of-Date Answers

Take a common pattern we see: an internal AI assistant keeps quoting a pricing structure that changed three months earlier. The bot wasn't malfunctioning. It was answering the question correctly, from its point of view: nobody had told it the pricing page had moved, and the copy of the pricing page it was working from was the one that existed the last time anyone rebuilt its index.

This is the pattern behind most of the "the AI gave a wrong answer" complaints we get called in to fix. The system isn't reasoning badly. It's reasoning correctly over old material and presenting the result with the same confidence it would use for something current. An AI system answers from what it can see, and if what it can see is a copy, the answer is only as current as that copy.

How does an AI assistant actually know anything?

Most business AI assistants don't query your live systems by default. They work from a retrieval setup: your documents, policies and knowledge base are chopped into pieces, converted into a searchable index, and when someone asks a question the system pulls the most relevant pieces and writes an answer from them. That index was built at some point in the past. It sits there, static, until something rebuilds it.

For genuinely stable material, this is the right architecture and there is nothing wrong with it. A staff handbook, a product manual, a set of terms and conditions: these change occasionally, on a schedule someone controls, and an index that is a few weeks old is still correct. The problem is that most businesses put everything into the same retrieval pipeline, including facts that are true for minutes rather than months, and the AI has no way of knowing the difference. It will answer a question about current stock levels with exactly the same tone of authority it uses for a question about your refund policy, because to the retrieval layer they are both just "text in the index".

When should an AI query live data instead of an index?

The decision rule is simpler than most teams make it. Ask how often the true answer to a given question changes. If the answer changes minute to minute or hour to hour (an account balance, a shipment status, whether a named contact still works at a company, current pricing on a live plan), the assistant has to call the system of record at the moment someone asks. Nothing else is trustworthy. If the answer changes over a quarter or a year (policy, process, product positioning, historical case studies), an index rebuilt on a sensible cadence is genuinely fine, and querying live data for it would be wasted engineering effort. Most failures we see come from teams who never drew this line at all. They built one retrieval pipeline, fed it everything, and assumed "up to date" was a property of the whole system rather than a property that has to be assigned fact by fact. Getting this right is largely a wiring decision: connecting an assistant to the actual system a fact lives in, rather than to a snapshot of it, is exactly the kind of work we cover in how to connect Claude directly to your HubSpot data, where the account record, deal stage and contact history are read live rather than from a copy that drifts out of sync with the CRM.

Where does retrieval quietly go wrong, even with a fresh index?

Assume, for the sake of argument, that the index is rebuilt often enough. Retrieval still breaks in three places that have nothing to do with freshness, and diagnosing a "wrong answer" complaint means checking all three before touching the refresh schedule.

How the documents were split into pieces. Long documents get cut into chunks before indexing, because a search system works on chunks, not whole documents. Cut a chunk at the wrong point and you separate a condition from the sentence it qualifies: "refunds are available within 30 days" ends up in one chunk, and "except for custom orders" ends up in the next one, never retrieved together. The answer that comes back is confidently wrong, and nobody thinks to blame the chunk boundary.

Whether the search actually matches the question. A customer asking "can I get my money back" and a document titled "Refund and Returns Policy" don't share many words in common, and a search built on keyword overlap alone can miss the connection entirely. Even search built on meaning rather than exact words retrieves the passages that are semantically closest to the question asked, which is not always the passage that actually answers it. The system pulls something that looks relevant and answers from that instead of admitting it found nothing solid.

How the retrieved pieces get assembled into an answer. Even with the right chunks retrieved, the model has to stitch several passages into one coherent response. If two retrieved chunks come from different versions of the same policy (an old one and its replacement, both still sitting in the document store), the model can blend them into an answer that was never true at any point in time. This is where a lot of the strangest-looking AI mistakes come from: not a hallucination in the usual sense, but a synthesis of two real, individually accurate fragments into a hybrid that is accurate to neither.

Why does an assistant get worse over time with no code changes?

The failures above are structural and show up from day one. There is a second, slower failure that shows up months in: a document store that grows faster than its index gets refreshed. Someone adds new policies, new product pages, new pricing sheets, and the index catches up on whatever cadence was set when the system first went live, usually weekly or monthly if it's scheduled at all, and often manually if it isn't. As the store grows and the refresh doesn't keep pace, the proportion of the knowledge base that is genuinely current keeps shrinking relative to the proportion that is stale, and the assistant's answers degrade gradually.

Nobody notices this as a single event, because there isn't one. Nobody deployed a bad change. The symptom, when it does surface, looks like the model "getting worse" or "less reliable", and teams often respond by tweaking prompts or trying a different model, when the actual cause is sitting in an index refresh job that was set up once and never revisited as the volume of source material grew. Catching this early is a monitoring problem as much as a data problem; our piece on the AI failures that do not crash covers what to instrument. If your assistant is built as an agent working across several connected tools rather than a single retrieval pipeline, the same principle still applies to each source it touches: an agent is only as current as the least-refreshed system it's wired into, a point covered in our overview of how Claude agents work across a business.

What is configuration decay, and how is it different?

There's a related failure that looks like a data problem but is actually an instruction problem. A standing instruction, prompt or template tells the assistant to format quotes a certain way, reference a specific document, or follow a particular escalation path. Six months later the knowledge base has moved on: the template was replaced, the escalation contact left, the referenced document was archived. The instruction was never wrong when it was written. It's wrong now, because everything around it changed and the instruction didn't. This is easy to miss because nobody thinks of a standing configuration as "content that goes stale", so it never appears on anyone's list of things to review. The fix is the same discipline as data freshness, just applied to instructions rather than facts: whatever tells the assistant how to behave needs the same ownership and review cycle as whatever tells it what to say.

What is the actual fix?

Three steps, done in order, and none of them are exotic.

• Classify every fact the assistant serves as either live-state or reference. Live-state is anything where the true answer can change before your next scheduled index rebuild. Reference is everything else. Do this fact by fact, not system by system: a CRM holds both stable company data and a live deal stage in the same record.
• Wire every live-state fact to the system of record, queried at the moment of asking. Not a nightly export, not a cached copy: a direct call. This is precisely what a controlled access layer between an assistant and your live systems is for, and it's the mechanism we set out in our guide to MCP integrations for Claude, where the assistant is given a defined, auditable way to reach specific tools rather than a snapshot of them.
• Put a named refresh owner and a review date on everything else. "The index refreshes automatically" is not an owner. Somebody has to be accountable for noticing when the document store has outgrown the refresh cadence, and somebody has to be accountable for re-reading standing instructions and templates on a schedule, not only when something visibly breaks.

None of this is about picking a smarter model. It's engineering: deciding what has to be live, building the connection that makes it live, and giving everything else an owner. That's the work behind most of what we build for clients, and it's the starting point of our own AI implementation service, which begins by mapping exactly this: what an assistant needs to answer from a live system, and what it can safely answer from a well-maintained index.

Frequently asked questions

How do I know if a fact should be live-state or reference?

Ask how long the true answer stays true. If it could be wrong within a day because someone changed a record, a price or a status, it's live-state and needs a direct query. If it only changes when someone deliberately rewrites a policy or a page, a periodically refreshed index is fine. When in doubt, treat it as live-state: the cost of an unnecessary live query is small, and the cost of a stale answer on something that matters is not.

Can I fix stale answers just by rebuilding the index more often?

Sometimes, but not always. A faster refresh schedule fixes slow-burn decay, where the document store has simply outgrown the old cadence. It does nothing for chunking errors, poor query matching or answers assembled from conflicting versions of the same document, because those are structural problems in how the retrieval pipeline is built, not timing problems. A more frequent rebuild of a badly chunked index still produces badly chunked answers, just more recently.

Does this apply to AI agents as well as simple chatbots?

It applies more, not less. An agent that takes action across several connected systems inherits the freshness problem of every system it touches, and a mistake compounds because the agent may act on a stale fact rather than merely state it. Wrong stock data quoted to a customer is a bad answer. Wrong stock data used to automatically confirm an order is a bad transaction.

Who should own the refresh schedule and review dates inside a business?

Whoever owns the underlying content should own its review date, the same as they would for a webpage or a policy document: the person responsible for pricing owns the pricing refresh, the person responsible for the returns policy owns that review. What should not happen is leaving it with whoever built the AI system, because they typically own the pipeline, not the judgement about which facts inside it have gone stale.

John Kelleher is a Claude Certified Architect (Foundations and Professional) and leads SpotDev, a Claude Registered Partner and OpenAI Select Partner.