Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models
- 1New York Eye and Ear Infirmary of Mount Sinai
Abstract
BackgroundA new class of AI "decision models" returns a probability for every option in a single pass and is marketed as calibrated, which invites use in clinical triage. Whether those probabilities hold up on clinical diagnosis has not been independently tested.
ObjectiveTo test the diagnostic accuracy, calibration, and probabilistic coherence of a decision model (Jev 1.13, TypeSafe AI) against frontier language models, with and without reasoning.
MethodsIn a pre-registered study (OSF, registered 26 September 2026), we compared Jev with Claude Opus 5.5 (adaptive reasoning, always on) and Claude Opus 5 (reasoning disabled) on three case sets: the 45 Semigran symptom-checker vignettes, 1,000 stratified synthetic patients from DDXPlus with reference differentials, and 80 simulated ophthalmic complaints (40 diagnoses at two levels of detail). Each model returned a probability for every diagnosis on a fixed list. Outcomes were top-1 and top-3 accuracy, expected calibration error (ECE), Brier score, the rate of exactly zero probability on the true diagnosis, sensitivity to option order, behavior on uninformative inputs, and behavior when the true diagnosis was absent.
ResultsOn DDXPlus, top-1 accuracy was 72.0% for Jev, 79.5% for Opus 5 without reasoning, and 87.4% for Opus 5.5 (both comparisons with Jev P<.001). Jev assigned exactly zero probability to the true diagnosis in 62 of 1,000 cases (6.2%); neither comparator did so in any case. Calibration error did not differ between Jev and the registered comparator (ECE difference 0.007, 95% CI -0.026 to 0.038). Offered an explicit "none of these" option, Jev declined to diagnose all 30 uninformative inputs; without that option, it assigned a mean 58% to a single diagnosis. Order sensitivity was small in all models. Given the option list with no patient information, Jev placed up to 96% on a single diagnosis, while the comparators spread probability almost evenly. On the ophthalmic complaints, accuracy did not differ (Jev 88.8%, Opus 5 91.2%, Opus 5.5 88.8%), and Jev was better calibrated than the registered comparator (ECE difference -0.081, 95% CI -0.124 to -0.018).
ConclusionsThe decision model matched frontier models on short, distinctive ophthalmic complaints but fell well behind on multi-symptom synthetic patients, including behind a comparator without reasoning, and there it excluded the correct diagnosis outright in about 1 in 16 cases. Its probabilities signaled uncertainty only when the question offered a way to abstain. Calibration claims for clinical decision models should be tested on clinical tasks, with explicit abstention options and zero-probability rates reported.
Related
- Non-diabetic endocrine neuropathies: A practical approach to diagnosis and clinical evaluation
- Molecular Diagnosis as a Probability of Necessity
- Homeostasis-Inspired, Quality-Adaptive Preprocessing for Multi-Label ECG Diagnosis: A Pre-Registered Multi-Objective Evolutionary Study with Negative Findings and Falsifiable Design Principles.
- Individualized and Platform-Designated Gene-Editing Studies in the Registered Clinical Record: A Case-by-Case Census Against 148 Eligible Studies
- Global clinical development of transcranial temporal interference stimulation: an analysis of registered human trials
BibTeX
@misc{deobhakta2026calibratedprobability,
title = {{Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models}},
author = {A. Deobhakta and J. S. Chang and R. W. S. Chen and S. Jaipalli and A. Pinhas and T. Y. T. Tai and R. Rosen},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.09.26365153},
url = {https://www.medrxiv.org/content/10.64898/2026.10.09.26365153v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models
AU - A. Deobhakta
AU - J. S. Chang
AU - R. W. S. Chen
AU - S. Jaipalli
AU - A. Pinhas
AU - T. Y. T. Tai
AU - R. Rosen
PY - 2026
DA - 2026/10/09
PB - medRxiv
DO - 10.64898/2026.10.09.26365153
UR - https://www.medrxiv.org/content/10.64898/2026.10.09.26365153v1
AB - BackgroundA new class of AI "decision models" returns a probability for every option in a single pass and is marketed as calibrated, which invites use in clinical triage. Whether those probabilities hold up on clinical diagnosis has not been independently tested. ObjectiveTo test the diagnostic accuracy, calibration, and probabilistic coherence of a decision model (Jev 1.13, TypeSafe AI) against frontier language models, with and without reasoning. MethodsIn a pre-registered study (OSF, registered 26 September 2026), we compared Jev with Claude Opus 5.5 (adaptive reasoning, always on) and Claude Opus 5 (reasoning disabled) on three case sets: the 45 Semigran symptom-checker vignettes, 1,000 stratified synthetic patients from DDXPlus with reference differentials, and 80 simulated ophthalmic complaints (40 diagnoses at two levels of detail). Each model returned a probability for every diagnosis on a fixed list. Outcomes were top-1 and top-3 accuracy, expected calibration error (ECE), Brier score, the rate of exactly zero probability on the true diagnosis, sensitivity to option order, behavior on uninformative inputs, and behavior when the true diagnosis was absent. ResultsOn DDXPlus, top-1 accuracy was 72.0% for Jev, 79.5% for Opus 5 without reasoning, and 87.4% for Opus 5.5 (both comparisons with Jev P<.001). Jev assigned exactly zero probability to the true diagnosis in 62 of 1,000 cases (6.2%); neither comparator did so in any case. Calibration error did not differ between Jev and the registered comparator (ECE difference 0.007, 95% CI -0.026 to 0.038). Offered an explicit "none of these" option, Jev declined to diagnose all 30 uninformative inputs; without that option, it assigned a mean 58% to a single diagnosis. Order sensitivity was small in all models. Given the option list with no patient information, Jev placed up to 96% on a single diagnosis, while the comparators spread probability almost evenly. On the ophthalmic complaints, accuracy did not differ (Jev 88.8%, Opus 5 91.2%, Opus 5.5 88.8%), and Jev was better calibrated than the registered comparator (ECE difference -0.081, 95% CI -0.124 to -0.018). ConclusionsThe decision model matched frontier models on short, distinctive ophthalmic complaints but fell well behind on multi-symptom synthetic patients, including behind a comparator without reasoning, and there it excluded the correct diagnosis outright in about 1 in 16 cases. Its probabilities signaled uncertainty only when the question offered a way to abstain. Calibration claims for clinical decision models should be tested on clinical tasks, with explicit abstention options and zero-probability rates reported.
ER -