Two complaints that sound alike but aren't
"It gave me the wrong answer" and "it couldn't tell me how it knew that" get lumped together as the same complaint: the chatbot isn't very good. They're not the same problem, and they don't carry the same risk.
A wrong answer is a mistake you can catch, once you notice it. An answer with nothing behind it, no document it points to, no specific detail it can stand on when you push, is a mistake you have no way to catch, whether it happens to be right or wrong this time.
What "can't explain itself" actually sounds like
It rarely shows up as a chatbot admitting it's guessing. It shows up as an answer that sounds complete the first time, then goes vague the moment you ask where it came from.
You: Can I get a refund after the 30 day window if the product was defective?
Chatbot: Yes, defective products are eligible for a refund even outside the standard 30 day window.
You: Where does it say that? I want to check before I request one.
Chatbot: Our policy is designed to be fair to customers who receive defective items, so exceptions like this are generally supported.
You: That's not really an answer. Which page or section says defective items are exempt from the window?
Chatbot: I don't have a specific section to point you to, but this is the kind of situation our policy is meant to cover.
Notice the shape of it. The first answer was confident and specific sounding, thirty days, defective products, an exception. The second and third answers get less specific each time, not more, because there was never a document behind the first one. It wasn't hiding a source. It didn't have one.
Why the unexplainable one is the riskier one
Being wrong sometimes is a rate you can live with, as long as you can tell which answers to double check. A chatbot that names a policy detail and a section you can go verify might occasionally be wrong, but you'll catch it, because you have something to check it against.
A chatbot that never gives you anything to check removes that safety net entirely, on every single answer, not just the wrong ones. You can't tell the correct answers from the confident-sounding ones by looking at them. Being right most of the time doesn't help if there's no way to tell which time is the exception.
You: Does the annual plan include the API access add-on, or is that separate?
Chatbot A: The annual plan includes API access as part of the base subscription. (Plan Comparison, updated March 2024)
You: That citation looks old. Is this still accurate?
Chatbot A: Fair point, that page is over a year old. Worth confirming with the team directly before you rely on it.
You: Does the annual plan include the API access add-on, or is that separate?
Chatbot B: Yes, API access is included with the annual plan.
You: How do you know that's current?
Chatbot B: That reflects our current plan structure.
Chatbot A handed you a real way to notice its answer might be stale. Chatbot B's answer might be completely accurate, or it might be outdated in exactly the same way, and there's no way to tell from inside the conversation. The first chatbot's visible flaw is also what makes it trustworthy. The second chatbot's confidence isn't evidence of anything.
Why this happens
A chatbot ending up unable to explain itself usually traces to one of a few ordinary causes, not to it trying to hide something:
There's no document behind the answer in the first place. If a chatbot is generating a plausible-sounding response instead of pulling from your actual content, there's nothing for it to point back to when you ask. It isn't withholding a source. There isn't one.
A citation exists, but not the part that actually supports the claim. Some chatbots link to a whole page as a nod toward transparency without connecting the specific sentence in that page to the specific claim in the answer. That's a citation in name, not something you can actually check a claim against.
The chatbot is tuned to sound confident, not to sound accurate. A support tool that hedges constantly reads as unhelpful, so some are built to smooth over gaps with assured-sounding language instead of a plain "I'm not sure." That produces answers that are consistently confident and inconsistently explainable.
What to test for
A few things worth trying yourself when you're evaluating a chatbot, since a vendor's own claims about its transparency are exactly what you're trying to verify:
Ask where it got the answer. Not "are you sure," which invites more confidence, but "which page or section says that." A chatbot with a real source behind it can usually get more specific when pushed. One without a real source tends to get vaguer, restating the claim in slightly different words instead of pointing anywhere.
Ask about something recent or easy to date. A chatbot that can tell you when its source was last updated, or admit it doesn't know, is giving you something to weigh. One that can't engage with the question of freshness at all is asking you to just trust it.
Notice whether pushing back produces a pointer or just more reassurance. "Here's the specific line that says that" is a pointer. "This is generally how it works" is reassurance. Only one of those actually helps you decide whether to trust the answer.
How we handle this
Our agent cites by article title, not with a vague gesture toward "your docs," so a claim worth double checking has an actual place to check it against. That a cited article is a real one it actually retrieved is enforced mechanically, not left to the model behaving well on any given answer. Whether every answer carries a citation in the first place is closer to an instruction it follows than a guarantee we can make; when it can't find anything in your documentation to support an answer, what it's told to do is say so plainly and point toward a real contact path rather than fill the gap with something confident sounding, and we test for that rather than promise it outright.
One exception: the chat box on our homepage runs on this site's own pages, a two-page corpus rather than a real help center, so it doesn't link citations there, since a two-page corpus would mostly just point you back to pages you can already see. That's a deliberate choice for a small demo corpus, not the behavior we build for an actual help center.
Accuracy and explainability are not the same test
Accuracy and explainability get evaluated together so often that it's easy to treat them as one thing. They're not. A chatbot's current accuracy rate is a snapshot you can't see from inside a single conversation. Whether it can show you where an answer came from is something you can check on the spot, on every question, before you've decided whether to trust any of it. That's the more useful thing to be testing for.
Related reading
- See all 20 guides in the Actually Helpful library, grouped by what you are actually trying to figure out.
- Why AI chatbots give confidently wrong answers, on how confidence and correctness get confused in the moment.
- How to tell if a chatbot is reading your docs or just guessing, on the behavioral signals that show up across a longer conversation.
- Why every chatbot answer should link to a real source, on what makes a citation worth trusting once it's there.
Ask it something it can't know
Ask the live agent something outside this site's own pages and read what happens. It runs on a two-page corpus, so it won't cite sources here, but you can see whether it says so plainly or covers the gap with something confident sounding.