Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals
Abstract
Performance uncertainty quantification is essential for reliable validation and eventual clinical translation of medical imaging artificial intelligence (AI). Confidence intervals (CIs) play a central role in this process by indicating how precise a reported performance estimate is. Yet, due to the limited amount of work examining CI behavior in medical imaging, the community remains largely unaware of how many diverse CI methods exist and how they behave in specific settings. The purpose of this study is to close this gap. To this end, we conducted a large-scale empirical analysis across a total of 24 segmentation and classification tasks, using 19 trained models per task group, a broad spectrum of commonly used performance metrics, multiple aggregation strategies, and several widely adopted CI methods. Reliability (coverage) and precision (width) of each CI method were estimated across all settings to characterize their dependence on study characteristics. Our analysis revealed five principal findings: 1) the sample size required for reliable CIs varies from a few dozens to several thousands of cases depending on study parameters; 2) CI behavior is strongly affected by the choice of performance metric; 3) aggregation strategy substantially influences the reliability of CIs, e.g. they require more observations for macro than for micro; 4) the machine learning problem (segmentation versus classification) modulates these effects; 5) different CI methods are not equally reliable and precise depending on the use case. Finally, we derived practical implications of this study in the form of a decision tree which shall prove useful to the community. This paves the way for future consensus guidelines on reporting performance uncertainty.
Related
- Confidence Intervals Based on the Modified Chi-Squared Distribution and Its Applications in Medicine
- Adaptive confidence intervals with missing data
- Changepoint localization via nested exchangeability tests: from point estimates to confidence intervals
- Sub-Gaussian Confidence Intervals for Heavy-Tailed Data: Characterizing the Limits of Inference
- Differentially Private Confidence Intervals for Relative Risk and Odds Ratio
BibTeX
@misc{andre2026performance,
title = {{Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals}},
author = {Pascaline André and Charles Heitz and Evangelia Christodoulou and Annika Reinke and Carole H Sudre and Michela Antonelli and Patrick Godau and M Jorge Cardoso and Antoine Gilson and Sophie Tezenas Du Montcel and Gaël Varoquaux and Lena Maier-Hein and Olivier Colliot},
howpublished = {HAL},
year = {2026},
month = sep,
url = {https://hal.science/hal-05472180v3},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals
AU - Pascaline André
AU - Charles Heitz
AU - Evangelia Christodoulou
AU - Annika Reinke
AU - Carole H Sudre
AU - Michela Antonelli
AU - Patrick Godau
AU - M Jorge Cardoso
AU - Antoine Gilson
AU - Sophie Tezenas Du Montcel
AU - Gaël Varoquaux
AU - Lena Maier-Hein
AU - Olivier Colliot
PY - 2026
DA - 2026/09/02
PB - HAL
UR - https://hal.science/hal-05472180v3
AB - Performance uncertainty quantification is essential for reliable validation and eventual clinical translation of medical imaging artificial intelligence (AI). Confidence intervals (CIs) play a central role in this process by indicating how precise a reported performance estimate is. Yet, due to the limited amount of work examining CI behavior in medical imaging, the community remains largely unaware of how many diverse CI methods exist and how they behave in specific settings. The purpose of this study is to close this gap. To this end, we conducted a large-scale empirical analysis across a total of 24 segmentation and classification tasks, using 19 trained models per task group, a broad spectrum of commonly used performance metrics, multiple aggregation strategies, and several widely adopted CI methods. Reliability (coverage) and precision (width) of each CI method were estimated across all settings to characterize their dependence on study characteristics. Our analysis revealed five principal findings: 1) the sample size required for reliable CIs varies from a few dozens to several thousands of cases depending on study parameters; 2) CI behavior is strongly affected by the choice of performance metric; 3) aggregation strategy substantially influences the reliability of CIs, e.g. they require more observations for macro than for micro; 4) the machine learning problem (segmentation versus classification) modulates these effects; 5) different CI methods are not equally reliable and precise depending on the use case. Finally, we derived practical implications of this study in the form of a decision tree which shall prove useful to the community. This paves the way for future consensus guidelines on reporting performance uncertainty.
ER -