Benchmarks
Every accuracy claim about SureCiteAI is traceable to a numbered run on this page. 6 benchmark suites. 297 test cases. Public methodology and raw run artifacts, published for independent audit.
Latest run: 2026-09-25T03:36:23Z · Configuration: RERANKER_PROVIDER=cohere · RAG_GROUNDING_PENALTY_ENABLED=true
Models that produced these figures: answers meta-llama/llama-3.3-70b-instruct, reranking Cohere rerank-3.5, query expansion bytedance-seed/seed-1.6-flash. Each of these has a newer model available. The figures describe this fleet. We publish the fleet because a benchmark without it is a number rather than a result — and because we measured a newer model and rejected it for fabricating a citation.
Why this page exists
Most RAG vendors quote percentages from internal evals on private data. Those numbers are unfalsifiable — there is nothing a buyer can independently verify. We made a deliberate choice the other way.
The numbers below are computed by our eval runner on golden cases derived from public corpora (CUAD, openFDA, SEC EDGAR, FinanceBench), against PDFs anyone can download from the original publishers. The methodology and every per-case result are published here so the numbers can be independently audited against those same public corpora. If a number is not on this page or in benchmarks/runs/, treat it as marketing copy, not measurement.
Latest scorecard
6 suites · 297 test cases · run on 2026-09-25T03:36:23Z.
What “pass” means. In this run a case passes when the expected source document is retrieved, the system answers or correctly declines, and every cited filename was in the retrieved context. It did not check that the answer matches the reference answer: a wrong figure quoted from the correct filing counts as a pass. Answer checking was added to the runner on 26 September 2026 and applies from the next published run.
| Suite | Pass | Retrieval hit | Hallucinations | ECE | License |
|---|---|---|---|---|---|
FinanceBench PatronusAI · 84 SEC filings · 32 issuers | 107/150(71%) | 140/150(93%) | 0/150 | 0.064 | CC-BY-NC-4.0 (aggregate scores fair-use citable) |
Legal (CUAD v1) Atticus Project · CC-BY-4.0 | 20/35(57%) | 19/27(70%) | 0/35 | 0.128 | Public |
Healthcare (openFDA) FDA drug labels · public domain | 35/35(100%) | 27/27(100%) | 0/35 | 0.255 | Public |
Accounting (SEC EDGAR) 10-K filings · public domain | 33/35(94%) | 23/23(100%) | 0/35 | 0.311 | Public |
Real Estate Internal corpus | 30/35(86%) | 20/20(100%) | 0/35 | 0.203 | Internal |
Consulting Internal demo | 7/7(100%) | 6/6(100%) | 0/7 | 0.059 | Internal |
| AGGREGATE | 232/297 (78%) | 235/253 (93%) | 0/297 | per suite | — |
What we measure
Retrieval hit-rate
Was the expected source document returned in the retrieved context? Scored only on cases where the question is answerable from the corpus.
Answer correctness
Not measured in the published run. Retrieval hit-rate says the right document was found, not that the answer drawn from it was right. The runner checks answer figures from 26 September 2026, and the next published run will include it.
Citation hallucination rate
Did the LLM cite a source filename that wasn't in the retrieved context? The single most damaging failure mode for a citation-first product. We hold this to zero by construction.
Abstention accuracy
Did the system correctly answer when grounded, and correctly refuse when the answer isn't in the corpus? Measured per-suite.
Calibration (ECE, Brier, AUROC)
Does the confidence score actually mean what it says? Lower ECE = better calibration. Computed per-suite — aggregating across suites is mathematically meaningless and we don't do it.
Reranker ablation
Three back-to-back runs of the same 147-case suite with only the reranker provider changed. This is how the per-tier reranker policy was decided.
| Provider | Tier | Pass rate | Hallucinations |
|---|---|---|---|
| None (hybrid only) | Trial | 122/147(83%) | 0/147 |
| Cohere rerank-3.5 | Enterprise / Custom | 126/147(86%) | 0/147 |
| Voyage rerank-2.5-lite | Paid (Solo / Team / Business / Scale) | 123/147(84%) | 1/147 (legal) |
Cohere is the enterprise default · Voyage Lite is the paid-tier default (~10× cheaper than Cohere) · Trial users get hybrid-only with the same zero hallucination guarantee.
How the runs are produced
These steps run in our private product repo. The eval runner and the CC-BY-NC FinanceBench goldens are not redistributable, so the exact run is not re-executable from the public repo — but the methodology and every raw per-case result are published there for independent audit:
# 1. Provision the eval tenants (idempotent). npx tsx scripts/eval/setup-eval-tenants.ts # 2. Pull the FinanceBench corpus (84 PDFs, ~12 min). npx tsx scripts/eval/sample-financebench.ts npx tsx scripts/eval/pull-financebench-corpus.ts # 3. Ingest into the per-suite Pinecone namespaces. npx tsx scripts/eval/ingest-eval-docs.ts --tenant eval-financebench # 4. Run the full suite (~50 min wall time). RERANKER_PROVIDER=cohere npm run eval:all # 5. Publish redacted artifact to benchmarks/runs/. npx tsx scripts/eval/publish-run.ts
References
- FinanceBench: A New Benchmark for Financial Question Answering — Patronus AI, 2023 (arXiv:2311.11944)
- CUAD: Contract Understanding Atticus Dataset — Hendrycks et al., NeurIPS 2021
- RAGAS: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023 (arXiv:2309.15217)
- openFDA — U.S. Food and Drug Administration
- SEC EDGAR — U.S. Securities and Exchange Commission
Want to verify these numbers on your own documents?
Start a 14-day free trial — no credit card required. Upload your contracts, workpapers, or filings and get cited answers in minutes.