The confidence problem: why chatbots rarely say "I don't know"

Why "I don't know" is the hardest thing for a chatbot to say

You: What happens if I cancel my subscription mid-month?

Chatbot: Your account will be closed immediately and you'll lose access to all features. A prorated refund will be issued within 5 business days.

You: I just checked the actual cancellation page. It doesn't say anything about a refund. Are you just making this up?

You caught the chatbot in a specific kind of trap, and it's not actually the chatbot's fault. The system generating that answer wasn't programmed to invent false refund policies. It was doing exactly what it was designed to do: attempt an answer to every question. And when a system is shaped by incentives to always attempt an answer, admitting "I don't know" becomes functionally impossible, even when it's the right thing to say.

The pull toward always answering something

Here's what happens in practice. A support chatbot gets a question. The natural output of a language model is text that continues from what it already has. That text might be wrong, but it's text. A person reading the answer usually doesn't have a way to know whether it came from your actual help docs or from the model's learned patterns about what plausible-sounding policy text looks like. Both read exactly like they belong there.

The system doesn't have an internal pressure gauge measuring certainty. It just generates the next most likely words. So if it has no good answer in the documentation, it still has an answer in the form of "here's what a similar policy might look like." That answer will sound confident either way.

Now imagine you're the person setting up this chatbot before it ships. You're testing it. It misses a question, or worse, it correctly says "I don't know" to something that seems like it should be answerable. What do you do? You tune it. You adjust the prompts. You add more documentation. You find ways to make it attempt more answers. During testing, a chatbot that refuses a lot looks broken. A chatbot that attempts answers looks smart, even when it's guessing. So you tune toward attempting, and that default sticks.

The honest trap in product setup

This happens at a level that has nothing to do with the technology. A wrong confident answer is indistinguishable from a right one in the moment you're testing the bot. A refusal feels like a failure mode. So the human incentive during setup pushes the same direction the technology naturally leans: toward always attempting an answer.

By the time the chatbot ships, "I don't know" is no longer the default behavior. It's the fallback that only triggers when the system genuinely has nothing plausible left to generate. And "genuinely has nothing" is a higher bar than "I actually don't have the right information" because the model can usually generate something plausible, even if it's wrong.

If you set this up yourself, you probably wanted it to admit gaps. But you were testing the behavior you could see in the moment, and in the moment, a refusal looks worse than a hedged guess. The wrong incentive during setup trains the wrong behavior into production.

What this looks like in the real world

Here's a concrete example. A SaaS platform has a feature that lets you export data to CSV. The documentation is clear about this. But a customer asks "Can I export to JSON?" A chatbot that has learned from thousands of examples of "export" and "CSV" and "JSON" might confidently generate: "Yes, you can export your data as JSON from the Data menu." It sounds authoritative. The customer tries it, can't find it, and now they're frustrated at both the product and the support experience.

The chatbot wasn't lying. It was doing what it was shaped to do: attempt an answer rather than admit a gap. A better answer would have been, "The export feature supports CSV format. I don't have documentation for JSON export, but our support team can check if that's in the product roadmap." That second answer is riskier during testing (it sounds less polished) and more honest (it separates what's documented from what might be possible).

The first answer wins during the tuning phase because it looks better to the person testing. The second answer only looks good once real customers are involved and something goes wrong.

A quick way to test a chatbot for this pattern

Ask it a question where the real answer is a documented "no" or "we don't support that."
Compare the chatbot's answer to what the actual help docs say. Look specifically for new details the docs don't mention.
If the bot adds specifics that improve the answer beyond what the docs contain, that's a red flag. An honest system pulls from what it has and stops.
Try a follow-up question that builds on the first answer. If it confidently continues a thread from an answer that may have been invented, you've found a system that's defaulting to "attempt something" over "admit a gap."

How we handle this

We tune our agent to close real gaps in what it's grounded in, not to guess past the ones that are still open. During our own testing, a refusal is not a failure mode. It's a correct output. When our agent doesn't find the right answer in your documentation, it says so and hands you off to someone who can help, rather than guessing at plausible-sounding text.

This means some test questions will come back with "I don't have that information." That reads as a limitation if you're only running through tests. But it's actually the right behavior, because the alternative is shipped guesses that become customer problems later. We measure success by whether an answer is grounded in your real docs, not by whether we attempted every question.

The incentive during setup is flipped on purpose. A chatbot that refuses a lot is doing its job. One that attempts everything is the one that worries us.