Accuracy Is Not Risk: Why Medical AI Should Not Report Only Its Most Likely Label
Abstract
Background. Many medical AI models are trained to reproduce a single reference label and hand the most likely label to the next step in a clinical workflow. A classifier can select that label correctly while providing an inadequate summary for estimating downstream risk. Methods. We use Jeffrey's rule of conditioning to specify when probabilities over intermediate states, such as diagnoses or imaging categories, can be carried forward to an outcome risk. We separate two errors of reporting only the most likely label, discarding the probability of the other states and applying the risk of patients who have the leading diagnosis to a patient who only probably has it. We also give conditions under which the label and the full distribution lead to the same decision, and we propose guardrails for device developers, regulatory reviewers, and health insurers. Results. In a hypothetical chest-pain differential, reporting only the benign leading diagnosis greatly understates the patient's risk of death or cardiac arrest, so a patient who warrants evaluation could be discharged or denied imaging. In a hypothetical BI-RADS example, the most likely category is "probably benign," yet the propagated malignancy risk exceeds the ceiling that defines that category. Conclusions. A predicted label is not an observed state, and a calibrated classifier is not automatically a valid outcome model. Clinical and insurance coverage decisions should rest on the patient's outcome risk, judged against a threshold justified by the consequences of acting and not acting. Developers should state what each model output means and check that every step between the model and the decision preserves the patient's risk.
Related
- Synthetic data for open medical research: what practitioners should know
- Relative risk reduction is not enough: why cardiorenal trials should routinely report absolute risk reduction and number needed to treat
- Likely and unlikely ramification of algebraic points on curves
- Scientist Mothers Working in Childcare Deserts Are Less Likely to Attain Tenure
- On the most likely geodesic in last passage percolation
BibTeX
@misc{humpherys2026accuracy,
title = {{Accuracy Is Not Risk: Why Medical AI Should Not Report Only Its Most Likely Label}},
author = {Jeffrey Humpherys and Makoto Jones},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01312},
eprint = {2611.01312},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01312},
}
RIS
TY - UNPB
TI - Accuracy Is Not Risk: Why Medical AI Should Not Report Only Its Most Likely Label
AU - Jeffrey Humpherys
AU - Makoto Jones
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01312
UR - https://arxiv.org/abs/2611.01312
AB - Background. Many medical AI models are trained to reproduce a single reference label and hand the most likely label to the next step in a clinical workflow. A classifier can select that label correctly while providing an inadequate summary for estimating downstream risk. Methods. We use Jeffrey's rule of conditioning to specify when probabilities over intermediate states, such as diagnoses or imaging categories, can be carried forward to an outcome risk. We separate two errors of reporting only the most likely label, discarding the probability of the other states and applying the risk of patients who have the leading diagnosis to a patient who only probably has it. We also give conditions under which the label and the full distribution lead to the same decision, and we propose guardrails for device developers, regulatory reviewers, and health insurers. Results. In a hypothetical chest-pain differential, reporting only the benign leading diagnosis greatly understates the patient's risk of death or cardiac arrest, so a patient who warrants evaluation could be discharged or denied imaging. In a hypothetical BI-RADS example, the most likely category is "probably benign," yet the propagated malignancy risk exceeds the ceiling that defines that category. Conclusions. A predicted label is not an observed state, and a calibrated classifier is not automatically a valid outcome model. Clinical and insurance coverage decisions should rest on the patient's outcome risk, judged against a threshold justified by the consequences of acting and not acting. Developers should state what each model output means and check that every step between the model and the decision preserves the patient's risk.
ER -