Spurious model comparisons are widespread in biomedical artificial intelligence
- 1National University of Singapore
Abstract
Cross-validation is routinely used to compare performance in biomedical artificial intelligence. Standard tests ignore correlation across cross-validation folds, inflating false-positive rates. In a PRISMA-guided meta-analysis of 184 studies (impact factor [≥] 15) across 30 biomedical fields, 97% use invalid tests. Among studies with abstract-level claims supported by invalid tests, 59% rely on a spurious comparison: one that loses significance after correcting for this correlation. On average, regaining significance requires a 43% larger effect. Extending these findings to the broader literature, we estimate that spurious comparisons occur in nearly two in three studies and support abstract-level claims in nearly one in three studies. Simulations confirm false-positive rates paradoxically approach 100% when cross-validation is repeated to improve stability. We introduce SHARP, a redesign of cross-validation, which best balances false-positive control and power among 13 benchmarked tests. These results reveal widespread fragility in biomedical artificial intelligence and offer a practical route to valid comparisons.
Related
- Toward Enactive Artificial Intelligence
- Growing up with artificial intelligence: An amplifier of inequalities and a public health issue
- Notification: Artificial Intelligence in Autism Spectrum Disorder Diagnosis: A Scoping Review of Face, Voice, and Text Analysis Methods.
- Early Detection of Red Palm Weevil Infestation: A Systematic Review of Sensing Technologies, Artificial Intelligence, and Field Readiness
- Kidney stones composition prediction using artificial intelligence ( KiSCAI) study
BibTeX
@misc{zeng2026spurious,
title = {{Spurious model comparisons are widespread in biomedical artificial intelligence}},
author = {T. Zeng and H. Li and S. Zhang and Y. Q. Tan and F. Tian and C. Orban and L. An and W. Che and J. Cheng and J. S. X. Chong and N. Dehestani and Z. Dong and X. Li and Z. Li and M. J. R. Lim and Y. Lin and Q. Ling and Z. Ling and X. Z. Low and S. Mansour L. and K. K. Ng and T. T. Nguyen and L. Q. R. Ooi and S. Pande and X. Qian and J. Ruan and Z. Wang and Y. Xie and C. Zhang and Y. Zhang and K. Patil and L. Parkes and E. Dhamala and S. Chopra and A. Zalesky and A. J. Holmes and S. Eickhoff and J. H. Zhou and O. Renaud and N. Dosenbach and K. P. Kording and D. Bzdok and T. Nichols and B. T. T. Yeo},
howpublished = {bioRxiv},
year = {2026},
month = may,
doi = {10.64898/2026.05.17.724301},
url = {https://www.biorxiv.org/content/10.64898/2026.05.17.724301v3},
note = {CC BY},
}
RIS
TY - UNPB
TI - Spurious model comparisons are widespread in biomedical artificial intelligence
AU - T. Zeng
AU - H. Li
AU - S. Zhang
AU - Y. Q. Tan
AU - F. Tian
AU - C. Orban
AU - L. An
AU - W. Che
AU - J. Cheng
AU - J. S. X. Chong
AU - N. Dehestani
AU - Z. Dong
AU - X. Li
AU - Z. Li
AU - M. J. R. Lim
AU - Y. Lin
AU - Q. Ling
AU - Z. Ling
AU - X. Z. Low
AU - S. Mansour L.
AU - K. K. Ng
AU - T. T. Nguyen
AU - L. Q. R. Ooi
AU - S. Pande
AU - X. Qian
AU - J. Ruan
AU - Z. Wang
AU - Y. Xie
AU - C. Zhang
AU - Y. Zhang
AU - K. Patil
AU - L. Parkes
AU - E. Dhamala
AU - S. Chopra
AU - A. Zalesky
AU - A. J. Holmes
AU - S. Eickhoff
AU - J. H. Zhou
AU - O. Renaud
AU - N. Dosenbach
AU - K. P. Kording
AU - D. Bzdok
AU - T. Nichols
AU - B. T. T. Yeo
PY - 2026
DA - 2026/05/17
PB - bioRxiv
DO - 10.64898/2026.05.17.724301
UR - https://www.biorxiv.org/content/10.64898/2026.05.17.724301v3
AB - Cross-validation is routinely used to compare performance in biomedical artificial intelligence. Standard tests ignore correlation across cross-validation folds, inflating false-positive rates. In a PRISMA-guided meta-analysis of 184 studies (impact factor [≥] 15) across 30 biomedical fields, 97% use invalid tests. Among studies with abstract-level claims supported by invalid tests, 59% rely on a spurious comparison: one that loses significance after correcting for this correlation. On average, regaining significance requires a 43% larger effect. Extending these findings to the broader literature, we estimate that spurious comparisons occur in nearly two in three studies and support abstract-level claims in nearly one in three studies. Simulations confirm false-positive rates paradoxically approach 100% when cross-validation is repeated to improve stability. We introduce SHARP, a redesign of cross-validation, which best balances false-positive control and power among 13 benchmarked tests. These results reveal widespread fragility in biomedical artificial intelligence and offer a practical route to valid comparisons.
ER -