Why demo accuracy and production accuracy differ
Demos use good documents. A clean PDF from a bank’s system extracts near-perfectly. A photograph of a physical form taken at a branch counter under a ceiling fan does not.
Production has a long tail. Handwriting, regional-language documents, stamps over text, multi-page files out of order, formats that changed last quarter.
Thresholds are risk decisions, not accuracy limits. We could raise the straight-through rate tomorrow by lowering the confidence threshold. In a regulated lending process that’s a terrible trade. The cost of one wrong field on a sanction letter exceeds the saving on a hundred files.
The exceptions are where the design lives
The mistake we see is teams designing the happy path and treating exceptions as a phase-two problem. Exceptions are the system.
What good exception design looks like: the file reaches a human with the source document already open at the right page, the specific disagreement highlighted, and the record it conflicts with alongside, so adjudication takes seconds rather than a fresh review.
What actually improves the number
In order of impact from our deployments:
The question to ask a vendor
Not “what’s your accuracy?” Anyone can answer that with a number from a benchmark you can’t inspect. Ask:
“Run it on two hundred of our actual files and show me the confidence distribution and the failure categories.”
If they won’t, that tells you what you needed to know. We’d expect you to ask us the same, and we’ll do it in the pilot.