Your AI Agent Doesn't Read Your Website. It Reads a Copy.

Here is a failure that is almost impossible to notice from the outside, and that we walked into ourselves last week.

A business corrects a claim on its website. The old page said one thing about who could use a paid feature; the new page says the opposite, plainly and correctly. The change goes live. Everyone moves on.

A customer then asks the AI support agent on that same website the same question — and gets the old answer. Not a vague answer. The wrong one, stated confidently, with the tone of something that had been checked.

Nothing is broken. No error is logged. The website is right, the support team is right, and the AI is wrong, and there is no alert anywhere because nothing failed.

Your AI agent does not read your website

The reason is simple once you see it, and almost nobody asks about it during a sales call.

An AI support agent does not answer from your live website. It answers from something derived from your website — a crawl, an index, a set of generated help articles, an embedding store. Something was read once, turned into a copy, and the copy is what the model sees.

That copy is the actual source of truth for every answer your customers receive. Your website is the source of truth for the copy. Those are different things, and the gap between them is measured in however long it takes for a re-crawl and a re-index to happen.

So "we fixed the page" and "the AI now gives the right answer" are two separate states. Fixing the first does not cause the second. It is worth saying that twice, because every instinct says otherwise: you corrected the source, so surely the thing reading the source is corrected too.

The bug that makes it permanent

The lag alone would be a scheduling problem. What turns it into a permanent wrong answer is a subtler thing, and it is the part we got wrong in our own product.

When a knowledge base is generated from a website, the generator has to decide what to do the second time it runs. Run it twice and you do not want two copies of every article. So you de-duplicate — if an article with this title already exists, skip it.

That is a completely reasonable rule. It is also, without anyone deciding it, a rule that says an existing article can never be updated.

The knowledge base becomes additive-only. New pages produce new articles; changed pages produce nothing at all. Every re-crawl looks successful, reports no errors, and changes nothing. The knowledge base cannot get more accurate — only larger and, on any page that changes, steadily more wrong.

A de-duplication rule and a freshness rule look like the same rule. They are opposites. "Do not create what already exists" had quietly become "never change what exists", and nothing in the logs could tell the difference between the two.

One claim, four places

When we went to fix a wrong answer, we found the real shape of the problem: a single claim was living in more places than we were editing.

The vendor's own pricing page said it. A generated help article said it. A "suggested question" produced by the same analysis presupposed it — the question itself assumed the capability, so even answering it honestly reinforced the premise. And the stored analysis from the crawl carried it too.

Correct three of those and the AI still gives the old answer, because the fourth one is the one it reads. We did exactly that, re-asked the agent, and got the wrong answer back — which is how we learned the mechanism rather than theorising about it.

The practical rule we now use: a corrected page is not a corrected answer. The only way to know whether a fix reached customers is to ask the agent the question again and read what it says. Not check that the page is right. Not check that the article is updated. Ask the thing that talks to customers, in the words a customer would use.

Why this is invisible to normal testing

Every check you would normally run passes.

The website returns a healthy response. The help article exists. The crawl completed. The AI answers promptly, fluently, and in your brand voice. There is no error state anywhere in the chain, because nothing errored — a stale copy is a perfectly valid copy of something that used to be true.

This is the same reason the failure survives audits. Someone reviews the help centre and finds the article present and well-written. Someone reviews the website and finds it accurate. Nobody compares the two on the specific sentence that changed, because nothing points at that sentence.

What to ask an AI helpdesk vendor

If you are evaluating AI support tools, these questions are cheap to ask and hard to answer vaguely. We would rather you asked us them too.

• When I correct a page on my site, what has to happen before the AI stops giving the old answer? If the answer is "it updates automatically", ask what triggers it and how often. "Automatically" usually means "on a schedule you cannot see".
• Can I trigger a re-index myself, without opening a support ticket? If a correction depends on someone else's cron job, your compliance position depends on their cron job.
• When a page changes, does the existing article get updated, or is a second one created? Duplicate articles are not a tidiness problem. Two articles that contradict each other are two grounding sources, and you do not control which one the model reaches for.
• Can I see when each article was last derived, and from which page? Without that, "is this current?" is unanswerable, and a stale article is indistinguishable from a fresh one.
• How do I verify a correction actually landed? The answer should involve asking the agent, not reading a dashboard.

What we changed

We hit every one of these, so it seems only fair to say what we did about them.

Re-analysis now updates an existing article in place rather than skipping it, keeping the same article so nothing linking to it breaks, and saving the previous version first so a wrong correction can be read back and undone. Identical content is left completely untouched, so re-running over an unchanged site is still a no-op.

Articles and suggested questions now record the page they came from. That turns "which page produced this, and has it changed?" from an archaeology exercise into a question you can answer directly.

And a re-index can now be triggered as a service action rather than requiring a human with the right login to press a button. That one matters more than it sounds: our own correction was blocked for a while because the only route to a re-crawl required a credential that had expired. A correction should never wait on someone remembering to log in.

The part that generalises

Most guidance about AI support agents is about the model — how good the answers sound, how often it resolves without a human. The failure that actually bites is duller and more mechanical: the model is only ever as current as the copy it reads, and nothing in a normal day tells you how old that copy is.

If you take one thing from this, take the test. After you correct something that matters — a price, an eligibility rule, a compliance position — go and ask your own AI agent about it, in a customer's words, and read the answer. It takes a minute. It is the only check in this whole chain that looks at what a customer actually receives.

DeskLeap is free — every feature, unlimited human agents. The only thing you ever pay for is an AI Agent at $29/month, with 500 AI replies included.