Tesseract 5.3.4, measured on 200 scanned retail receipts, and the review threshold its own numbers actually support.
This is a real measurement, not a worked example. The subject is a real, widely deployed OCR engine reading real scanned receipts, against ground truth published with the corpus. No customer data is involved and nothing here is simulated: every figure below came out of the run, including the ones that are inconvenient. The corpus is public and the code is a few hundred lines, so any of it can be reproduced or contradicted.
The subject matter is the set of confidence scores a document-extraction system reports alongside each value it reads, and whether those scores are calibrated — that is, whether the values it scores 0.9 are correct about 90% of the time.
Calibration matters because it is what a human-review threshold rests on. A system that routes low-confidence extractions to a reviewer is only as safe as the score doing the routing. Where that score is an artefact of the model's output layer rather than a probability, the routing is decorative.
Why this subject. Tesseract is open, in production in a very large number of pipelines, and reports a per-word confidence that engineers threshold on. Auditing it requires nobody's private data and costs nobody a customer relationship. It is the cheapest available answer to a fair question: does this measurement actually show anything, or only work on data chosen to make it work?
In a commissioned engagement, the system owner is responsible for the extraction system, the confidence scores it emits, and the correctness labels supplied. The auditor is responsible for measuring the relationship between the two and reporting what it shows.
This report differs in one respect, and it is worth stating plainly. Nobody commissioned it. The labels come from a published academic corpus rather than from a system owner's review queue, and the auditor chose both the subject and the corpus. That makes this weaker evidence about Tesseract-in-your-pipeline than a commissioned audit would be, and it is why section 6 says what it says about representativeness.
Standard, published methods, applied so that every figure in this report can be independently reproduced from the same inputs:
One prediction is one annotated text region: the region is cropped from the scan, the engine reads it, and the read is scored right or wrong against the transcript published with the corpus. Localisation is given, so what is measured is recognition confidence — the number a production system thresholds on — and not the separate question of whether the region was found.
No language model takes part in the measurement. Every figure in this report is arithmetic over a table of confidences and correctness labels — bucketing, counting, and a Wilson interval. Nothing is scored by a model, summarised by a model or judged by a model. Where a value is compared with ground truth the comparison is an exact string match, with only case and whitespace forgiven; no system decides whether two values are close enough. This matters because an audit whose verdict rested on a model's judgment would fail in exactly the way the audit exists to detect — sharing its blind spots with the system under test, and reporting a flattering number with confidence.
The confidence scores do not mean what they say. Every band from 0.2 upward claims more than it delivers, and the gap peaks near the middle: values scored around 0.75 are correct 40% of the time.
At a 95% accuracy target there is no threshold on this data at which automated decisions reach the target at the lower bound. Under this comparison rule, the honest recommendation is that this system should not be automating against that target at all.
0.90 is the threshold teams pick when nobody has measured anything. 5,088 of the 10,720 regions — 47% of all traffic — score at or above it, and 714 of those are wrong. That is a 14.0% error rate inside the band chosen precisely because it was supposed to be safe to leave alone.
The individual failures are worse than the rate. Among values shaped like money — digits, a decimal point, two more digits — 1,010 score at or above 0.90 and 136 of those are wrong. Roughly a quarter of those errors are a lost decimal point.
| Confidence | On the receipt | Read as | Factor |
|---|---|---|---|
| 0.96 | 9.00 | 900 | ×100 |
| 0.96 | 4.60 | 460 | ×100 |
| 0.96 | 4.77 | 477 | ×100 |
| 0.96 | 2.25 | 225 | ×100 |
| 0.96 | 4.30 | 430 | ×100 |
| 0.96 | 2.0 | 20 | ×10 |
These are not near misses. A dropped decimal point is a hundredfold error on a monetary amount, and it arrives at the confidence level at which a pipeline stops looking. An aggregate accuracy figure of 60.7% does not tell you this is the shape of the failure; only the list does.
| Band | n | Claims | Delivers | Gap | 95% CI |
|---|---|---|---|---|---|
| 0.0–0.1 | 241 | 0.019 | 0.112 | -0.093 | 0.078–0.158 |
| 0.1–0.2 | 115 | 0.147 | 0.139 | +0.008 | 0.087–0.214 |
| 0.2–0.3 | 155 | 0.250 | 0.168 | +0.083 | 0.117–0.234 |
| 0.3–0.4 | 238 | 0.351 | 0.143 | +0.208 | 0.104–0.193 |
| 0.4–0.5 | 338 | 0.449 | 0.201 | +0.248 | 0.162–0.247 |
| 0.5–0.6 | 549 | 0.550 | 0.273 | +0.277 | 0.238–0.312 |
| 0.6–0.7 | 790 | 0.651 | 0.297 | +0.353 | 0.267–0.330 |
| 0.7–0.8 | 1,187 | 0.749 | 0.400 | +0.349 | 0.373–0.428 |
| 0.8–0.9 | 2,019 | 0.853 | 0.544 | +0.310 | 0.522–0.565 |
| 0.9–1.0 | 5,088 | 0.940 | 0.860 | +0.080 | 0.850–0.869 |
The two heaviest bands are marked. Every band from 0.2 up overclaims by more than sampling noise. The worst gap is not at the top but in the middle — the 0.6–0.7 band claims 0.651 and delivers 0.297 — while the top band, carrying 47% of all traffic, claims 0.940 and delivers 0.860. The lowest band underclaims: things scored near zero are right 11% of the time.
Under this report's rule — exact match, only case and whitespace forgiven — no threshold reaches 95% accuracy at the lower bound. There is no number to recommend.
Relax the rule to also forgive punctuation at the start and end of a value, which is defensible when the value is a field rather than a line of prose, and a threshold appears: ≥ 0.96, automating 11.5% of volume at 96.9% accuracy, 95.8% at the lower bound. Same engine, same documents, same day.
That difference is the finding, not a footnote to it. The looser rule changes overall accuracy from 60.7% to 64.7% and the calibration error from 0.191 to 0.155 — and it changes the recommendation from do not automate to automate an eighth of it.
So an accuracy figure quoted without its comparison rule is not a measurement. Vendors publish headline accuracy constantly and almost never publish the rule underneath it. Both numbers above are honest; they are not the same number; and nothing but the rule separates them. This report states its rule in section 3 and reports the sensitivity here, which is the least that should be expected of any figure of this kind.
Note also what the 0.96 threshold buys even when it works: 88.5% of the documents still go to a human. A threshold that meets the target by automating almost nothing is a true answer to the wrong question, and it should be read as evidence about the engine rather than as a recommendation to deploy it.
| Content | n | ECE | Accuracy |
|---|---|---|---|
| mixed (letters and digits) | 2,682 | 0.314 | 45.2% |
| numeric | 3,280 | 0.159 | 63.1% |
| alphabetic | 4,615 | 0.146 | 68.7% |
| symbolic | 143 | 0.139 | 35.7% |
Mixed content — dates, reference numbers, amounts carrying a currency code, anything combining letters and digits — is more than twice as miscalibrated as plain prose and correct less than half the time. These are the fields document automation exists to capture. Confidence is least trustworthy exactly where the decisions are made, and an overall figure of 0.191 conceals a 0.314 inside it.
Everything above describes this sample. Whether it describes production is a separate question, and it is the one most capable of making a rigorous report worthless.
Every statistical claim in sections 4 and 5 quantifies sampling error — how far the observed accuracy might sit from the truth for the population this sample was drawn from. None of them can detect that the sample came from the wrong population. If 200 documents are hand-picked, an immaculate report can be produced about traffic that does not exist.
| Content | n | Share of sample | Production share |
|---|---|---|---|
| alphabetic | 4,615 | 43.1% | unknown |
| numeric | 3,280 | 30.6% | unknown |
| mixed | 2,682 | 25.0% | unknown |
| symbolic | 143 | 1.3% | unknown |
Representativeness: UNVERIFIED — and it stays that way. These are Malaysian retail receipts from a 2019 competition corpus. Nobody has stated a production mix for them because there is no production system here, so the audit reports the verdict it is entitled to rather than the one that would read better. The findings describe this sample. Whether they describe any particular pipeline is not something this report can establish, and a commissioned engagement would begin by fixing exactly that.
Three things this check enforces, and they are refusals rather than reassurances:
Calibration is a comparison between what a system claimed and what turned out to be true. The second half of that sentence is not free. Every figure in this report inherits the quality of the correctness labels it was measured against, and those labels are the least examined part of most evaluations.
Here they are the transcripts published with the corpus, produced by its annotators. In a commissioned engagement they come from the system owner instead — an existing review queue, extractions a person has already checked in the course of normal work. They are not commissioned for the audit.
The auditor does not generate ground truth, and in particular does not generate it with a language model. This is a refusal, not an omission. A model asked to judge another model's extraction fails on the same documents — the same smudged scan, the same ambiguous field — and it fails in the same direction. Its errors would be correlated with the errors under test, which does not merely add noise: it cancels out precisely the disagreements that reveal overconfidence. The system would be reported as better calibrated than it is, and the report would be most wrong exactly where it mattered most. There is no sample size that repairs this.
And these labels are not clean, which is the point of saying where they came from. One region is transcribed SDN BND; the engine read SDN BHD, which is the correct Malaysian company suffix and almost certainly what is printed. The audit scored the engine wrong. That is one visible instance of a class of error nobody has counted here, and it runs in the direction of making the engine look worse than it is.
| Recorded | Why it is in the report |
|---|---|
| Source | Which review queue or process the labels came from, named. |
| Method | Whether a person checked the field against the document, or a downstream correction was inferred. |
| Re-check | Whether a subset was independently re-labelled, and how often the two disagreed. |
| Floor | The disagreement rate above, carried into every conclusion as a limit on resolution. |
The floor is the point of the table, and this report does not have one. No independent re-check of the corpus transcripts was performed, so the label error rate here is unquantified and every figure above should be read as resting on it. If a re-check found the labels disagreed 3% of the time, no calibration gap smaller than roughly three percentage points could be distinguished from label noise — which would leave the central findings standing, since they are an order of magnitude larger, and would rightly dissolve the small ones.
This work is performed independently. The author has no commercial relationship with the Tesseract project, no interest in the corpus, and no fee contingent on the conclusion reached. Tesseract was selected because auditing it requires nobody's private data.
This is a measurement, not a certification. It is not an assurance engagement conducted under ISAE 3000 or any equivalent standard, and no accreditation, professional body membership or regulatory approval is claimed or implied. It is an independent measurement, reported plainly, that you can reproduce yourself.
The author sells this measurement as a service, which is a reason to check the method rather than take the conclusion on trust. A commissioned report is prepared for the system owner and is not intended for any other party. This one carries no such restriction — it exists to be checked.
Five document-extraction vendors publish a rule for when to trust a confidence score. None publish evidence that the score is calibrated.