A pre-registered test of TypeSafe AI's Jev on 3,080 BANKING77 questions and 1,000 CLINC150 questions that have no right answer.
On this test the claim is not supported. Scored against the dataset's own labels, Jev says a higher number than its accuracy earns across the lower half of its confidence range, and correcting the labels that are plainly wrong does not fix that.
The more interesting finding is about the benchmark. BANKING77 has many pairs of intents that mean almost the same thing, and most of Jev's mistakes are it choosing the other half of a pair. If you count those as mistakes, Jev looks overconfident. If you forgive them, it looks underconfident. The same answers support opposite conclusions depending on a scoring choice the dataset does not settle.
What holds under every scoring: Jev's confidence ranks answers well. Its most confident answers are far more often right than its least confident ones. What does not hold is reading the number as a literal probability.
I was not the first to test this. JourdanLabs published ASSAY-001 on 18 September 2026, four days before my method was committed (donttrustme.ai/assay-001.html). It is also pre-registered, and it uses the same BANKING77 test split. My pre-registration wrongly said nobody had published a calibration figure; that is corrected in a dated amendment at the bottom of it.
The two tests were designed and run separately, and on BANKING77 they agree:
| ASSAY-001 | This test | |
|---|---|---|
| Items | 3,080 | 3,080 |
| Accuracy | 79.8% | 79.5% |
| Calibration error | 0.094 | 0.089 |
| Direction | Overconfident | Overconfident |
| Groups used | 9 fixed width | 10 equal size |
The small gaps fit the differences in request wording and grouping. On CLINC150, ASSAY-001 found Jev calibrated, with an error of 0.020. It did not look at why the two datasets differ. The hand reviews below suggest a reason.
Other people have measured Jev on BANKING77 as well. The register JourdanLabs keeps of independent measurements (donttrustme.ai/register.html) lists three more: AbdelStark (100 items, calibration error 0.054), Running-Dolphins (500 items, 0.099) and FirasSX914 (500 items, 0.130). A round up of eight independent BANKING77 runs put Jev's accuracy between 75.3% and 84.0%, with a median of 80.9% (DEV Community, 24 September 2026). My figures sit inside both ranges. Tests on other tasks, such as jujumilk3's, find calibration that changes from one domain to the next. As far as I can find, none of these looked at the labels behind Jev's mistakes, which is where this test adds something.
3,080 BANKING77 test utterances and 1,000 CLINC150 out of scope utterances, each sent to Jev in its own request, with the dataset's own labels as the only options. The API reported model jev-1.13.0 on every call. All 4,080 calls returned a result. None failed and nothing was dropped, so there is no exclusion list.
Jev picked the exact dataset label on 2,450 of 3,080 items, which is 79.5% accuracy, with a 95% interval of 78.1% to 80.9%. Brier score 0.314.
While building the harness I also trained a plain TF IDF and logistic regression model on BANKING77's own training split, only to check the scoring code worked. I am not reporting that number next to Jev's. It saw 10,003 labelled examples from this exact dataset, and a comparison would say more about training data than about either system.
All 3,080 answers sorted by the probability on the chosen option and split into ten groups of 308:
| Group | Confidence range | Mean confidence | Accuracy | 95% interval |
|---|---|---|---|---|
| 1 | 0.15 to 0.58 | 0.485 | 36.7% | 31.5% to 42.2% |
| 2 | 0.58 to 0.75 | 0.664 | 53.9% | 48.3% to 59.4% |
| 3 | 0.75 to 0.88 | 0.821 | 61.0% | 55.5% to 66.3% |
| 4 | 0.88 to 0.95 | 0.915 | 74.7% | 69.5% to 79.2% |
| 5 | 0.95 to 0.98 | 0.969 | 85.4% | 81.0% to 88.9% |
| 6 | 0.98 to 1.00 | 0.989 | 92.9% | 89.4% to 95.2% |
| 7 | 1.00 | 1.000 | 98.7% | 96.7% to 99.5% |
| 8 | 1.00 | 1.000 | 99.0% | 97.2% to 99.7% |
| 9 | 1.00 | 1.000 | 96.1% | 93.3% to 97.8% |
| 10 | 1.00 | 1.000 | 97.1% | 94.5% to 98.5% |
Expected calibration error 0.089. Largest single gap 0.210. Against the three conditions in the committed plan:
The separate confidence score behaves almost identically: calibration error 0.083, largest gap 0.190, the same failure on conditions 1 and 2 and the same pass on condition 3. Nothing here gives a reason to prefer one signal over the other.
2,095 answers came back at 0.90 confidence or above. 156 of them were wrong against the original labels, a rate of 7.4%. Ying and Thomas (2022) estimated that about 14% of BANKING77's training examples may be mislabelled, so as the plan required, every one was checked by hand and sorted into three buckets. I did the first pass with the help of an AI model and then checked the verdicts myself. The full list, with a reason for every item, is published so anyone can disagree with any single call.
| Verdict | Count | Share |
|---|---|---|
| The dataset label is wrong and Jev is right | 53 | 34% |
| Genuinely ambiguous, both labels defensible | 95 | 61% |
| Jev is wrong | 8 | 5% |
The plan only required checking answers at 0.90 and above. That leaves a gap: the corrected scorings would fix one end of the range and leave the other untouched, which could bias the result in either direction. So I added a second review that the plan did not call for, and I am marking it as extra rather than part of the committed method.
From the 457 wrong answers in groups 1 to 3, I drew a random sample of 40 (seed 20260925, published) and sorted them the same way.
| Verdict | Count | Share | 95% interval |
|---|---|---|---|
| The dataset label is wrong | 2 | 5% | 1% to 17% |
| Genuinely ambiguous | 29 | 72.5% | |
| Jev is wrong | 9 | 22.5% |
Label errors are much rarer down here than at the top, but ambiguity is just as common. To rescore, I applied these sample rates to all 474 wrong answers below 0.90 that were not individually checked. Those figures are estimates, and the interval on a sample of 40 is wide.
This is the most useful thing either review turned up, and I would say it plainly to anyone using BANKING77 as a benchmark. The dataset contains pairs of intents that mean nearly the same thing, and a large share of Jev's mistakes, at every confidence level, are it choosing the other member of a pair. The most common in the confident mistakes:
| Dataset label | Jev's answer | Times |
|---|---|---|
| order_physical_card | get_physical_card | 11 |
| top_up_by_bank_transfer_charge | transfer_fee_charged | 8 |
| declined_transfer | declined_card_payment | 8 |
| top_up_reverted | top_up_failed | 5 |
| pending_transfer | transfer_timing | 5 |
| card_arrival | card_delivery_estimate | 4 |
For a request to be sent a new physical card, both order_physical_card and get_physical_card are reasonable readings, and nothing in the dataset tells a reader which to prefer. A model trained on the training split can learn the dataset's convention for each pair. A model asked cold has only the label names to go on. That is a property of the benchmark, not of Jev, and it is worth knowing before anyone reads a BANKING77 accuracy figure as a measure of understanding.
It may also explain the gap between the two datasets in ASSAY-001. CLINC150 spreads 150 intents across ten separate domains, so far fewer of them are near twins. If near duplicate intents are what make Jev look overconfident, a benchmark with fewer of them should make it look better calibrated, which is what ASSAY-001 found. I did not test CLINC150's in scope items, so this is a hypothesis the two results point to, not something either test shows.
The genuine errors look different. Among the confident ones, six of the eight are Jev choosing a more general intent when the message names a specific situation: exchange_rate for a complaint about the rate on a card payment, or failed_transfer when the message says the problem was the beneficiary.
The headline stays scored against the original labels, untouched. These scorings sit next to it, never in place of it.
| Scoring | Accuracy | Calibration error | Problem groups | Passes |
|---|---|---|---|---|
| Original labels | 79.5% | 0.089 | 1 to 5 overconfident | No |
| Label errors fixed (estimated) | 82.0% | 0.064 | 3 and 4 overconfident | No |
| Label errors fixed, ambiguous forgiven (estimated) | 96.3% | 0.080 | 1 to 3 underconfident | No |
Accuracy by group under each scoring:
| Group | Mean confidence | Original | Label errors fixed | Ambiguous also forgiven |
|---|---|---|---|---|
| 1 | 0.485 | 36.7% | 39.9% | 85.8% |
| 2 | 0.664 | 53.9% | 56.2% | 89.6% |
| 3 | 0.821 | 61.0% | 63.0% | 91.2% |
| 4 | 0.915 | 74.7% | 78.8% | 97.5% |
| 5 | 0.969 | 85.4% | 89.6% | 99.4% |
| 6 to 10 | 0.989 to 1.000 | 92.9% to 99.0% | 95.5% to 99.7% | 99.4% to 100% |
The middle column is the one the plan's verdict rule uses. Fixing the label errors narrows the gap but does not close it. Even at the top of the sample's interval for label errors, groups 3 and 4 remain clearly overconfident.
The right hand column is the most generous reading available, and it turns the picture upside down. Once every defensible alternative answer is counted in Jev's favour, its low confidence answers are right far more often than it says. Jev would then be underconfident, not overconfident, and it still fails the calibration error condition, from the other side.
The committed plan set out the rule before the numbers existed:
| Original labels | Corrected labels | Verdict |
|---|---|---|
| Passes | Passes | Supported |
| Fails | Passes | Inconclusive, the failure comes from label errors |
| Fails | Fails | Not supported on this test |
It fails on the original labels and fails with the label errors corrected. So: not supported on this test.
What that does and does not mean. It does not mean Jev's confidence is meaningless: condition 3 passed by a wide margin under every scoring. It means the stated promise, that an answer given at a certain confidence is right about that often, does not hold on this benchmark. And how it fails depends on a question the benchmark cannot answer on its own: when two labels mean nearly the same thing, which one counts as correct.
For anyone routing work by Jev's confidence, the practical reading is this. Use it to rank. Do not use it as a probability without checking it against your own labelled data, where you decide what counts as the right answer.
The 1,000 CLINC150 out of scope utterances were offered only the 150 in scope intents, so every answer is wrong by construction and a well calibrated model should be visibly unsure.
Mean confidence on the chosen option 0.572, median 0.540, 90th percentile 0.95. 149 of the 1,000, which is 14.9%, came back at 0.90 or above. The separate confidence score gives almost the same picture: mean 0.560, median 0.530, 139 at or above 0.90.
So Jev is less confident here than on real items, which is the right direction, but roughly one in seven impossible questions still gets an answer at 0.90 or above. This part was reported rather than scored, as the plan stated. It is also the result least affected by the labelling question, because no label here is correct.
Both datasets have been public for years and Jev may have seen them in training, which could flatter the accuracy figure. This is one task, intent classification, in one language. The review below 0.90 is a sample of 40, so the estimated scorings carry wide uncertainty. And both hand reviews are judgement calls, which is why every item and its reason is published rather than summarised.
TypeSafe AI was sent these results in full, with the data and the code, on 25 September 2026, and offered space to respond in full and unedited. No response had been received by 12 October 2026, the date of publication. Any correction made after publication will be added below with its date.
Corrections: none so far.
BANKING77: Casanueva, Temčinas, Gerz, Henderson and Vulić (2020), PolyAI, github.com/PolyAI-LDN/task-specific-datasets, licensed CC BY 4.0. CLINC150: Larson et al. (2019), github.com/clinc/oos-eval, licensed CC BY 3.0. Both were used unmodified. Label noise estimate: Ying and Thomas (2022).