Confidence thresholds in document extraction: when a field goes to review.
Document extraction rarely fails all at once. It produces mostly correct fields, a few wrong ones and a score for each. The operational question is which fields a person should check. A good threshold answers that per field, rests on measured errors rather than on the score's label, and keeps the review workload visible.
Published
A confidence score is not a probability until you check it
Extraction models and OCR engines return scores, and large language models can be asked to state their confidence. None of these numbers means that a field marked 0.9 is correct nine times out of ten. Scores are useful for ranking fields from more to less likely to be wrong; how they relate to actual error rates has to be measured on your own documents.
To measure it, run extraction on documents with checked answers, group the fields by score and record the observed error rate in each group. The result shows where errors concentrate and whether the score separates good fields from bad ones at all.
Set thresholds per field, not per document
An invoice total and a free-text reference on the same invoice have different consequences when wrong and different error profiles. One threshold for the whole document either sends too many harmless fields to review or lets expensive errors through. Set each field's threshold from two numbers: what a wrong value costs downstream and what a review costs in time.
Route the uncertain field, not the whole document. A reviewer who sees the extracted value, the source region and the reason it was flagged resolves it far faster than one who has to re-read the entire document.
Use validation rules before scores
Where a field can be checked, a check is more reliable than a score. Many fields carry their own evidence of correctness or can be compared with data you already hold.
- Check digits and formats: IBANs, Swedish organisation numbers and many other identifiers include a checksum.
- Arithmetic: line items should sum to the subtotal, and subtotal plus tax to the total.
- Reference data: a supplier, customer or product should exist in the master data it will be matched against.
- Plausibility: dates within an expected range and amounts in the usual order of magnitude for that counterparty.
A field that passes validation with a moderate score may be safe to accept. A field that fails validation should go to review whatever its score.
Measure the errors you accept, not only the ones you catch
Once thresholds are in place, fields above them flow through without review and their errors become invisible. Check a regular sample of auto-accepted fields. That sample is the only direct measure of the residual error rate, and it is what tells you whether a threshold is still right.
Report the review rate alongside it. Lower thresholds reduce review work and raise residual errors; the right setting is a trade-off the process owner agrees to with both numbers in front of them.
Expect the inputs to change
New suppliers, new layouts, poor scans and changes to the extraction model all move the relationship between score and error. Track review rate and sampled error per document type over time, and recalibrate against fresh checked answers when they drift. Store every reviewer correction with its source reference; those corrections are the cheapest evaluation data you will have.
What to hand over
Thresholds without their evidence get changed by guesswork. Hand over the reasoning together with the numbers.
- Field definitions, validation rules and each field's threshold, with the measurements behind it.
- The calibration set and the procedure for repeating the calibration.
- The review queue design, including what the reviewer sees and how corrections are stored.
- Reports on review volume, sampled residual error and change by document type.
Where Landvex fits
Our document processing work evaluates extraction field by field on your documents and hands over the thresholds together with the measurements behind them. Read how we scope and deliver document processing.
Bring one concrete workflow
Describe the work as it happens today: who does it, how often, and what breaks. You will hear back from a founder.