Clinical Prediction Model Performance Metrics Cluster at Arbitrary Thresholds and are Incorrectly Reported in High-Ranking Journals: An Observational Meta-Research Study
- 1Queensland University of Technology
Abstract
Background Previous research has shown that clinical prediction models have likely been 'hacked' to meet arbitrary performance thresholds. However, this work only assessed abstracts, could not identify models with a perfect AUC (area under the receiver operating characteristic curve) of 1, and did not examine other performance metrics. Methods We collected data from full texts and validated a large language model to extract data from a random sample of Q1 journal articles for 2025. We examined if hacking occurred in the AUC value, sensitivity and specificity, and checked if articles correctly reported the AUC value from corresponding receiver operating characteristic (ROC) curves. Results We identified 2,001 prediction model articles that reported 15,656 AUC values. We found a striking excess of results near perfect prediction and at multiples of five and ten for the AUC, sensitivity and specificity, indicating likely manipulated data, methods or publication bias. Around 1 in 40 AUC values were reported outside of their confidence intervals, and around 1 in 8 ROC curves did not match their reported AUC value. Conclusion We have shown that not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice. Excess models that are not representative of their true performance may misguide clinicians, distort the literature, and make it harder to find prediction models that are useful for practice.
Related
- Transfusion practices, thresholds, and complications in pediatric oncology patients: a single-center prospective longitudinal observational study
- Group normative data and arbitrary asymmetry thresholds fail to represent individual players in weekly hip strength monitoring
- Prediction of Spinal Pain Intensity from Perception-Dependent Stimulation Thresholds and Patient Characteristics: A Regression and Machine Learning Approach
- Clinical performance and decision thresholds of an automated plasma p-tau217 assay for amyloid pathology in a Taiwanese memory clinic.
- Peer Evaluation Rating Behavior in Differentiating Student Performance in Undergraduate Medical Education: An Observational Study
BibTeX
@misc{gibson2026clinical,
title = {{Clinical Prediction Model Performance Metrics Cluster at Arbitrary Thresholds and are Incorrectly Reported in High-Ranking Journals: An Observational Meta-Research Study}},
author = {A. D. Gibson and D. N. Borg and N. M. White and A. Barnett},
howpublished = {medRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.02.26367135},
url = {https://www.medrxiv.org/content/10.64898/2026.11.02.26367135v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Clinical Prediction Model Performance Metrics Cluster at Arbitrary Thresholds and are Incorrectly Reported in High-Ranking Journals: An Observational Meta-Research Study
AU - A. D. Gibson
AU - D. N. Borg
AU - N. M. White
AU - A. Barnett
PY - 2026
DA - 2026/11/02
PB - medRxiv
DO - 10.64898/2026.11.02.26367135
UR - https://www.medrxiv.org/content/10.64898/2026.11.02.26367135v1
AB - Background Previous research has shown that clinical prediction models have likely been 'hacked' to meet arbitrary performance thresholds. However, this work only assessed abstracts, could not identify models with a perfect AUC (area under the receiver operating characteristic curve) of 1, and did not examine other performance metrics. Methods We collected data from full texts and validated a large language model to extract data from a random sample of Q1 journal articles for 2025. We examined if hacking occurred in the AUC value, sensitivity and specificity, and checked if articles correctly reported the AUC value from corresponding receiver operating characteristic (ROC) curves. Results We identified 2,001 prediction model articles that reported 15,656 AUC values. We found a striking excess of results near perfect prediction and at multiples of five and ten for the AUC, sensitivity and specificity, indicating likely manipulated data, methods or publication bias. Around 1 in 40 AUC values were reported outside of their confidence intervals, and around 1 in 8 ROC curves did not match their reported AUC value. Conclusion We have shown that not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice. Excess models that are not representative of their true performance may misguide clinicians, distort the literature, and make it harder to find prediction models that are useful for practice.
ER -