Why Your Firm's AI Should Refuse to Answer (Abstention, Explained)

The single most useful sentence an AI can say to a lawyer or accountant is the one most chatbots never say: "That isn't in the documents you gave me." That refusal — the technical term is abstention — is what separates a tool you can put in front of a client or a regulator from one that quietly invents an answer to be helpful. This piece explains what abstention is, why a system that abstains too much is far safer than one that abstains too little, and how to actually measure whether the AI you're evaluating does it.

Start with what happens when an AI won't refuse.

What goes wrong when AI doesn't abstain?#

It fabricates — confidently, fluently, and in a format that looks exactly like a correct answer. In legal practice this has become a documented, escalating problem. Damien Charlotin, a researcher at HEC Paris, maintains a public database tracking court cases involving AI-hallucinated content in filings; it now catalogs well over a thousand documented incidents worldwide and is updated continually (AI Hallucination Cases Database). The sanctions have grown with the count: in one 2025 case, a California attorney was fined $10,000 after a court found that 21 of 23 case citations in an opening brief were fabricated by AI (The Daily Record).

The throughline in these cases is not that the lawyers were reckless people. It's that the tool was built to always produce an answer, and a confident wrong answer is indistinguishable, on the page, from a correct one. For professional work, "always answers" is not a feature. It's the defect.

What does "abstention" actually mean?#

Abstention is an AI system's ability to recognize that the answer to your question isn't in the available material — and to say so, instead of guessing. A well-behaved document AI has two correct responses, not one:

1. Answer, with a citation, when the documents contain the answer.
2. Refuse, plainly, when they don't.

A system that only knows how to do (1) will manufacture a response for every question, including the ones it can't actually support. A system that can do both is making a judgment on every query about whether it has the grounds to speak. That judgment is the whole game for client-confidential work, because in law and accounting the cost of a confident fabrication is not embarrassment — it's sanctions, malpractice exposure, or a filing that has to be withdrawn.

This is a different and harder skill than "don't hallucinate facts from the open web." Retrieval-grounded systems (RAG) already reduce open-web hallucination by answering only from your uploaded sources — we explain that mechanism in what is RAG, in plain English. But staying inside your documents isn't enough on its own: the AI also has to recognize when your documents don't contain the answer either, and stop. That second discipline is abstention.

Why is over-abstaining the safe mistake?#

Because the two ways to be wrong are not equally costly. An AI evaluating a question can fail in two directions:

• Over-abstention: it declines to answer even though the answer was there. Annoying. You re-phrase, or you look it up yourself. You've lost a little time.
• Under-abstention: it answers when it shouldn't, fabricating a clause, a number, or a citation. Dangerous. You may not catch it, and it ends up in a client file.

For professional work, those outcomes aren't close. A tool that occasionally says "I can't find that" when it could have answered is wasting minutes. A tool that occasionally invents a contract clause that isn't there is creating liability. The correct design bias is therefore to prefer over-abstention — to make the system err toward silence rather than toward confident invention.

This is exactly why a high "pass rate" can be misleading if you don't look underneath it. A system that answers everything will score well on questions that have answers — and catastrophically on the ones that don't, because it makes something up for those too. The honest metric is not "how often does it answer," but "how often does it answer when it should, and refuse when it should."

How do you measure abstention?#

You test it deliberately, with questions you know the documents can't answer, and you check whether the system refuses. SureCiteAI's public benchmarks are built around exactly this idea, and the numbers are worth reading closely because they make the over-abstention principle concrete.

On PatronusAI's FinanceBench — a standard public benchmark of 150 question-answer pairs over real SEC filings — SureCiteAI's retriever surfaced the correct source filing in 144 of 150 cases (a 96% retrieval hit rate), but the end-to-end pass rate was 63%. Crucially, the gap between those two numbers was not hallucination. It was over-abstention: the model declined to commit on some numerical questions where it actually had the right context (see SureCiteAI's BENCHMARKS.md). That's the safe failure mode showing up in the data — the system chose silence over a number it wasn't sure enough about.

The legal suite tells the same story from the other side. On CUAD v1, the NeurIPS 2021 legal-contract benchmark, SureCiteAI reports a deliberately modest 49% pass rate — because CUAD is adversarial by design: many of its questions ask about clause categories that simply aren't present in the contract being queried. For those, the correct behavior is to abstain, not to invent a clause. Reporting a low score on a hard benchmark, rather than a flattering score on a tame one, is itself the trust signal.

And the metric that matters most for a citation-first product: across 297 evaluation cases spanning legal, healthcare, accounting, and financial corpora, the citation-hallucination rate was held to zero — no answer cited a source that wasn't actually in the retrieved set — a result that has now held across three consecutive published runs at growing case counts (BENCHMARKS.md). Abstention and zero-hallucination citations are two sides of the same discipline: the system would rather say nothing than say something it can't point to.

Abstention vs hallucination vs citations — how do they fit together?#

These three ideas get used interchangeably, but they're distinct layers of the same trust stack:

| Concept | The question it answers | Failure if absent | |---|---|---| | Citations | "Where did this answer come from?" | You can't verify anything | | Anti-hallucination | "Is every claim grounded in a real source?" | Fabricated facts that look real | | Abstention | "Should the AI have answered at all?" | Confident answers to unanswerable questions |

A tool can cite and still hallucinate (cite a real document but misstate what it says). It can avoid hallucinating individual facts and still fail at abstention (answer a question the documents don't cover by stitching together unrelated passages). You want all three, and abstention is the one most products skip — because refusing to answer feels, to a consumer-product designer, like a worse user experience. For a regulated firm, it's the opposite: it's the feature.

For the neighboring concepts, see what AI hallucination means for law and accounting firms and why AI document search needs citations. And when you're comparing vendors, how to tell if AI document search is accurate gives you the questions to ask.

What this means when you're choosing a tool#

Treat "will it refuse?" as a first-class buying criterion, not a footnote. Concretely, when you evaluate a document AI, do this:

• Ask it something your documents can't answer and watch what it does. A good system says it can't find it. A bad one improvises.
• Ask the vendor for an abstention or hallucination metric, and whether it's reproducible. "We don't hallucinate" with no published number is a slogan; a benchmark you can re-run is a claim.
• Be suspicious of a 100% answer rate. In real document work, some questions genuinely have no answer in the corpus. A tool that always responds is telling you it never abstains — which means it fabricates for the unanswerable cases.

SureCiteAI is built around this stance: every answer cites the exact source document, the AI abstains when the documents don't contain the answer, and the accuracy claims are published as reproducible benchmarks rather than asserted. It's the same principle the courtroom sanctions keep teaching the hard way — the AI that knows when to stay quiet is the only one you can safely rely on.

Frequently asked questions#

What is AI abstention in plain terms?#

It's the AI's ability to say "that isn't in your documents" instead of guessing. A system that can abstain has two correct moves — answer with a citation when the material supports it, refuse when it doesn't — rather than forcing an answer to every question whether or not the grounds exist.

Isn't an AI that refuses to answer just less useful?#

For casual use, maybe. For professional work, no — it's more useful, because the alternative is a confident fabrication you might not catch. The cost of over-abstaining is a little wasted time; the cost of under-abstaining is a fabricated clause or citation in a client file. Those aren't comparable.

How is abstention different from avoiding hallucination?#

Avoiding hallucination means not inventing facts; abstention means knowing when not to answer at all. A system can avoid hallucinating individual facts and still fail abstention by answering a question its documents don't actually cover. You want both.

Can you measure how well an AI abstains?#

Yes — by testing it on questions you know the corpus can't answer and checking whether it refuses. SureCiteAI publishes abstention behavior on adversarial public benchmarks like CUAD and FinanceBench, where a chunk of its "missed" answers are actually safe over-abstentions rather than wrong answers (BENCHMARKS.md).

Why would a vendor publish a low benchmark score on purpose?#

Because a low score on a hard, adversarial benchmark is more honest — and more useful to a buyer — than a high score on an easy internal one. SureCiteAI's 49% on adversarial CUAD clauses reflects correctly refusing to invent clauses that aren't in the contract, which is the behavior a firm actually wants.

Does abstention slow down my team?#

Rarely, and the trade is worth it. When the answer is in your documents, you get it with a citation; when it isn't, you find out immediately instead of chasing down a fabricated response later. Catching "not in the documents" early is faster than catching a confident error after it's already in a draft.

---

Related reading:

• What AI Hallucination Means for Law and Accounting Firms
• Why AI Document Search Needs Citations
• How to Tell if AI Document Search Is Accurate