AI systems attach a confidence score to their answers, and software uses it to decide what a person checks and what goes straight through. This index records what each maker says about that score, and whether anyone has published evidence that it holds.
Calibration evidence means data showing how often answers at a given confidence turn out to be right, such as a reliability curve or a calibration error figure. An accuracy benchmark does not count: it says how often a system is right overall, not whether its confidence tells you which answers to trust.
| System | What the maker says | Evidence published | Independent measurement | Checked |
|---|---|---|---|---|
| JevTypeSafe AI | Says it is a probability of being rightCalibrated: higher confidence means higher accuracy.typesafe.ai |
None found | PublishedKorbant test (12 Oct 2026, pre-registered): calibration error 0.089 on BANKING77, overconfident; the benchmark’s near-duplicate intents decide the direction. Agrees with ASSAY-001 by JourdanLabs (18 Sep 2026): 0.094 on BANKING77, 0.020 on CLINC150. | 12 Oct 2026 |
| Strands Decider 2BAWS Strands Labs (open source) | Says it is a probability of being rightCalibrated reliability scores: every decision carries a confidence.github.com/strands-labs |
Published by makerSelf-reported on JevBench v1: calibration error 0.052; answers at a confidence of 0.9 or more are right about 95% of the timeon unseen short classification tasks. Not yet checked independently. |
In progressKorbant test planned on the same datasets as Jev. | 3 Oct 2026 |
| Microsoft-Decision-1Microsoft | Says it is a probability of being rightIt returns a calibrated probability for each option instead of generated text.ai.azure.com |
None foundThe model card gives no calibration metric. A launch chart shows a “calibration” score of 92.2 without saying how it is measured. | Not measured | 11 Oct 2026 |
| Tesseract 5.3.4Open source OCR | No claim madeNo calibration claim. Reports a confidence for each word it reads.github.com/tesseract-ocr | No claim made | PublishedCalibration error 0.191 on 200 scanned receipts. | 3 Sep 2026 |
| Amazon TextractAmazon Web Services | Says it is a probability of being rightA confidence score is a number between 0 and 100 that indicates the probability that a given prediction is correct.docs.aws.amazon.com |
None found | Not measured | 25 Sep 2026 |
| Azure AI Document IntelligenceMicrosoft | Says it is a probability of being rightFor example, a confidence value of 0.95 (95%) indicates that the prediction is likely correct 19 out of 20 times.learn.microsoft.com |
None found | Not measured | 25 Sep 2026 |
| RossumRossum | Says it is a probability of being rightWe calibrate the confidence scores to correspond to the probability of a given value being right.Default automation threshold 0.975.knowledge-base.rossum.ai |
None foundCalibration is claimed. No figures are published. | Not measured | 25 Sep 2026 |
| LlamaExtractLlamaIndex | Says it is a probability of being rightOn some tiers, a score approximates a real probability of correctness.Two tiers are described as not calibrated.developers.llamaindex.ai |
None foundOne figure: at 0.8, roughly 75% of errors fall below the line. No curve or calibration error. | Not measured | 25 Sep 2026 |
| Landing AIDocument parsing | Says not to read it as a probabilityRecommends 0.95 as a starting threshold, and says the score is not a probability of correctness.landing.ai |
None found | Not measured | 25 Sep 2026 |
| HyperscienceHyperscience | Says not to read it as a probabilityNote that machine confidence is different from accuracy and probability.help.hyperscience.ai |
None found | Not measured | 25 Sep 2026 |
| MindeeMindee | Says not to read it as a probabilityThe confidence score in Automation is a consensus-based reliability measure, not a simple probability.docs.mindee.com |
None found | Not measured | 25 Sep 2026 |
| Google Document AIGoogle Cloud | Thresholds or ranking onlyThe confidence score communicates how strongly your model associates each entity with the predicted value.docs.cloud.google.com |
None found | Not measured | 25 Sep 2026 |
| NanonetsNanonets | Thresholds or ranking onlyA higher confidence score represents a higher probability that the model is correctly predicting a field. Recommends a threshold of about 0.8.docs.nanonets.com |
None foundSays 0.8 typically representsprecision of 0.9, in our experience. No data is published. |
Not measured | 25 Sep 2026 |
| VeryfiVeryfi | Thresholds or ranking onlyThe minimum recommended threshold is between 0.7 and 0.9, depending on the strictness of your use case, but don’t rely on it blindlyfaq.veryfi.com |
None found | Not measured | 25 Sep 2026 |
| RunpulseDocument extraction | Thresholds or ranking onlyAuto accept at 0.95 and above, audit a 5% sample from 0.85, review below 0.85, reject below 0.50. States a target calibration error below 0.02.runpulse.com | None foundA target is stated. No measured value is published. | Not measured | 25 Sep 2026 |
| ExtendDocument extraction | Thresholds or ranking onlyUse them to trust high-confidence values automatically and route the rest to human review.docs.extend.ai |
None found | Not measured | 25 Sep 2026 |
| AffindaDocument extraction | Thresholds or ranking onlyField thresholds set as validation rules, for example confidence is >80%. Automatic validation against a global threshold is available on request.docs.affinda.com |
None found | Not measured | 25 Sep 2026 |
| DocsumoDocument extraction | Thresholds or ranking onlyThe confidence score is a valuable indicator that informs you about the system’s level of confidence in the accuracy of extracted field values.support.docsumo.com |
None found | Not measured | 25 Sep 2026 |
| InfrrdDocument extraction | Thresholds or ranking onlyConfidence Score is the level of certainty or reliability associated with the extracted data.infrrd.ai |
None found | Not measured | 25 Sep 2026 |
| InstabaseDocument extraction | Thresholds or ranking onlyNon-OCR confidence is calculated using log probabilities, which provide a mathematical measure of how likely a result is.docs.instabase.com |
None foundSays the method proves reliablein internal testing. No data is published. |
Not measured | 25 Sep 2026 |
| ReductoDocument extraction | Thresholds or ranking onlyLow parse_confidence suggests OCR errors. Low extract_confidence suggests the model was uncertain about interpretation.docs.reducto.ai |
None found | Not measured | 25 Sep 2026 |
| SensibleDocument extraction | Thresholds or ranking onlyReturns uncertainty signals reported by the language model, and warns: Note that LLMs can inaccurately report confidence signals.docs.sensible.so |
None found | Not measured | 25 Sep 2026 |
| super.AIDocument processing | Thresholds or ranking onlyAuto accept and auto reject boundaries: below one a job needs review, above the other it is marked correct. Listed as coming soon.docs.super.ai |
None found | Not measured | 25 Sep 2026 |
| KlearStackDocument extraction | Thresholds or ranking onlyLow-confidence fields are routed to human-in-the-loop validation workflows.No numeric threshold published.klearstack.com |
None found | Not measured | 25 Sep 2026 |
| ABBYY VantageABBYY | Thresholds or ranking onlyDropped its field confidence value: it always contained a fixed constant numeric value 1.0.Points users to flagged characters instead.support.abbyy.com |
None found | Not measured | 25 Sep 2026 |
| Einstein ClassificationSalesforce | Says it is a probability of being rightPrediction confidence measures the likelihood that Einstein’s best prediction for the field value is correct.help.salesforce.com |
None found | Not measured | 25 Sep 2026 |
| Amazon Lex V2Amazon Web Services | Thresholds or ranking onlyThe confidence scores that Amazon Lex V2 returns are comparative values. You should not rely on them as an absolute score.docs.aws.amazon.com |
None found | Not measured | 25 Sep 2026 |
| Dialogflow CXGoogle Cloud | Thresholds or ranking onlyIf the highest scoring intent has a confidence score greater than or equal to the classification threshold setting, it is returned as a match.docs.cloud.google.com |
None found | Not measured | 25 Sep 2026 |
| Azure AI Language CLUMicrosoft | Thresholds or ranking onlySet the score according to your own observations of prediction scores because they might vary by project.learn.microsoft.com |
None found | Not measured | 25 Sep 2026 |
| watsonx AssistantIBM | Thresholds or ranking onlyIntents below 0.2 stop triggering: the dialog node stops being processed.cloud.ibm.com |
None found | Not measured | 25 Sep 2026 |
| RasaRasa | Thresholds or ranking onlythe intent nlu_fallback will be predicted when all other intent predictions fall below the configured confidence threshold.legacy-docs-oss.rasa.com |
None found | Not measured | 25 Sep 2026 |
| Cohere ClassifyCohere | Thresholds or ranking onlyThe confidence level represents the model’s degree of certainty that the query falls under a given label.docs.cohere.com |
None found | Not measured | 25 Sep 2026 |
Corrections and replies: bilal@korbant.co.uk
Request a measurement. Make an AI system that shows a confidence score? You can ask to have it measured. The method is pre-registered, the result is published whatever it shows, and you get the same right of reply as everyone else. bilal@korbant.co.uk
Tesseract 5.3.4 on 200 scanned receipts, and the review threshold its own numbers support.
Five document extraction vendors publish a rule for when to trust a score. None publish evidence that it holds.
A pre-registered test of TypeSafe AI's Jev. Not supported on BANKING77, and the benchmark's near-duplicate intents decide which way it misses.