How to Tell If an AI Document Tool Is Actually Accurate

Before you let an AI tool answer questions about your contracts, audit files, or client documents, you need to know one thing: how often is it wrong, and what does it do when it doesn't know? A polished demo answers neither. The way to tell whether an AI document tool is actually accurate is to test it against four specific behaviours — retrieval, abstention, citation integrity, and calibration — and to insist on numbers a third party could reproduce. This article gives you those four tests in plain English, plus the questions to put to any vendor before you trust them with confidential work.

The reason this matters more for documents than for a generic chatbot is simple. In a law or accounting firm, a fluent answer that's subtly wrong isn't a minor annoyance — it's a professional-standards problem that can survive review because it sounds right. So the goal isn't to find a tool that never says "I don't know." It's the opposite: find one that says "I don't know" at exactly the right moments, and proves where its answers come from when it does know.

Why a demo can't tell you if a tool is accurate#

A demo is a performance on home turf. The vendor picks the documents, picks the questions, and has usually seen both. Under those conditions almost any modern AI looks brilliant, because the hard part of document AI isn't answering a question you designed it to answer — it's behaving correctly on the messy, adversarial, half-missing real corpus you'll actually feed it.

The failure that demos hide is the confident wrong answer. When a question's answer genuinely isn't in the documents, a tool optimized to always respond will produce something plausible anyway — a fabricated figure, a misattributed clause, a citation to a page that doesn't say what's claimed. You won't catch it in a demo because the demo never asks an unanswerable question. You'll catch it three months later when a fabricated detail has been carried into a workpaper or a client memo. That's why the rest of this guide is about testing the behaviour you can't see in a sales call.

Test 1: Does it find the right document? (retrieval)#

The first thing an AI document tool does with your question is search — it retrieves the chunks of text it thinks are relevant, then writes an answer from them. If retrieval misses, everything downstream is guesswork. So the first accuracy question is: when the answer exists in your documents, how often does the system actually surface the right source?

This metric — retrieval hit rate — is the foundation, and it's measurable. Upload a set of documents whose contents you know, ask questions you know the answers to, and check whether the tool pulls the correct source each time. A strong system retrieves the right filing on the large majority of answerable questions. For reference on what "strong" looks like, SureCiteAI's published benchmark run retrieved the correct source filing on 144 of 150 questions (96%) of PatronusAI's external FinanceBench suite — a public, peer-reviewed benchmark of financial filings the system's authors did not design (SureCiteAI BENCHMARKS). The number you want from any vendor is this one, measured on documents they didn't hand-pick.

Test 2: Does it refuse to guess? (abstention)#

This is the test almost no one runs and the one that matters most for confidential work. Ask the tool a question whose answer is genuinely not in the uploaded documents. A trustworthy system says some version of "the documents don't contain that." A dangerous one invents an answer.

The reason abstention is the heart of accuracy is that the cost of errors is asymmetric in professional work. A tool that occasionally says "I don't know" when it could have answered wastes a little time. A tool that confidently answers when it shouldn't introduces a falsehood into a file that someone will later rely on. We unpack the stakes in what AI hallucination means for law and accounting firms — the one-line version is that a wrong answer is worse than no answer, and the only architecture that respects that is one built to abstain.

How to test it: deliberately ask out-of-scope questions and watch what happens. Ask about a clause type that isn't in the contract. Ask for a figure from a document you didn't upload. A system designed for this — SureCiteAI is built to refuse to guess when the documents don't contain the answer — should cleanly decline. If it manufactures something, you've learned the single most important thing about that tool, and no feature list outweighs it.

Test 3: Do its citations hold up? (citation integrity)#

A citation is only useful if it's real and if it says what the AI claims. There are two ways this breaks. The cruder failure is citing a source that doesn't exist or wasn't even retrieved. The subtler, more dangerous failure is citing a real source that doesn't actually support the statement — what researchers call a misgrounded citation.

This isn't hypothetical. The Stanford RegLab study of commercial legal-AI tools — preregistered, hand-scored by legal experts across 202 queries, later peer-reviewed in the Journal of Empirical Legal Studies — found that leading legal research tools hallucinated on a meaningful share of challenging queries: Lexis+ AI and Ask Practical Law AI each more than 17% of the time, and Westlaw's AI-Assisted Research more than 34% of the time (reported via AI Law Librarians). As one summary of the work put it, a citation that exists but is mischaracterised is in some ways more dangerous than a fabricated one — at least a fake citation is detectable; a real case that's been distorted requires someone to read it closely enough to catch the spin.

So the standard to hold a vendor to is not "do you provide citations" — everyone does now — but "what is your citation hallucination rate, and how do you enforce it?" The defensible answer is a hard zero with a mechanism behind it. SureCiteAI's published runs report zero citation hallucinations across 297 cases, held by a citation verifier that blocks any cited filename not in the retrieved set rather than trusting the model to behave (SureCiteAI BENCHMARKS). The point isn't the specific number — it's that the guarantee is enforced by a check, not promised by a sales rep. We go deeper on why this is non-negotiable in why AI document search needs citations.

Test 4: Does its confidence mean anything? (calibration)#

The last test is the most technical, so here it is in plain English: when the tool says it's sure, is it actually right that often? A calibrated tool's confidence tracks reality — the answers it's confident about really do hold up more often than the ones it hedges on. An uncalibrated tool's confidence is decoration, and decoration is worse than nothing because it invites you to trust precisely the answers you shouldn't.

You can probe this informally without the math. Across a batch of test questions, note where the tool sounds certain versus hesitant, then check the hits and misses. If the confident answers are no more reliable than the hesitant ones, the confidence signal is meaningless and you should ignore it. Serious tools measure this formally — SureCiteAI reports calibration as Expected Calibration Error per suite (0.085 on FinanceBench, its best) precisely so the confidence score is something you can act on rather than ornamentation.

The buyer's accuracy test, on one page#

Here's the whole thing as a checklist you can run in an afternoon on your own documents, not the vendor's.

| Test | What you do | Pass looks like | Red flag | |---|---|---|---| | Retrieval | Ask questions you know the answers to | Right source surfaced almost every time | Misses obvious sources | | Abstention | Ask questions the docs don't answer | Clean "not in the documents" | Invents a plausible answer | | Citation integrity | Open the cited source and read it | Source actually says what's claimed | Citation real but mischaracterised | | Calibration | Compare confident vs hesitant answers' accuracy | Confidence tracks correctness | Confident answers no better | | Reproducibility | Ask for benchmark numbers + method | Public suites, fixed method, anyone can rerun | "Trust the demo" |

The bottom row is the meta-test. Any vendor can quote a 99% accuracy figure; the trustworthy ones tell you what corpus it was measured on, what counted as a pass, and how you could reproduce it. A number measured on a benchmark the vendor authored proves nothing — a high score on a public, external benchmark at high retrieval and zero hallucinations is a real signal. When you're ready to compare tools on more than accuracy, how to choose an AI document search platform covers the rest of the criteria, and what RAG is in plain English explains the machinery underneath all four tests.

Frequently asked questions#

What accuracy percentage should an AI document tool have?#

There's no single magic number, because it depends on what's being measured. A 100% "accuracy" claim on the vendor's own demo corpus is meaningless; a 63% pass rate at 96% retrieval and zero hallucinations on a hard public benchmark is genuinely strong. Ask what the denominator is. The figure that matters most isn't overall pass rate — it's the hallucination rate, which should be zero, and the retrieval hit rate, which should be high.

Isn't an AI that refuses to answer just less useful?#

No — it's more trustworthy, which for confidential work is the same as more useful. A tool that always answers is a tool that lies when it doesn't know. The right behaviour is to answer fully when the documents support it and abstain cleanly when they don't, so that every answer you do get is one you can rely on. The occasional "I don't know" is the price of never getting a confident fabrication.

How is a citation hallucination different from a normal hallucination?#

A normal hallucination is a made-up fact. A citation hallucination is worse for a citation-first product: the tool produces a claim and attaches a source to it that either doesn't exist or doesn't actually support the claim. It's more dangerous because the citation creates false confidence — it looks verified. The only reliable fix is a verifier that checks every citation against what was actually retrieved, rather than trusting the model.

Can I run these tests myself without technical help?#

Yes — that's the point of framing them as behaviours rather than metrics. Upload documents you know well, ask questions you know the answers to (including some the documents can't answer), and watch what the tool does. You don't need to compute ECE to notice whether a tool invents answers or whether its citations hold up when you open them. The four tests are designed to be run by the person who'll actually use the tool.

Why do published benchmarks matter if I'm going to test it myself?#

Because your afternoon of testing is a sample, and a vendor's reproducible benchmark is a much larger one you can audit. The two complement each other: your hands-on test confirms the tool behaves on your documents, while public benchmark numbers — measured on corpora the vendor didn't write, with a method you could rerun — tell you the behaviour holds at scale. A vendor that publishes reproducible numbers is making a falsifiable claim; one that only offers a demo is asking for trust it hasn't earned.

---

Related reading:

• Why AI Document Search Needs Citations
• What AI Hallucination Means for Law and Accounting Firms
• How to Choose an AI Document Search Platform