Cohort Versus Nested Case-Control Design for Prediction Models of Rare Outcomes Using Electronic Health Records
Abstract
ObjectiveTo determine how cohort versus nested case-control (NCC) design affects EHR-based prediction of a rare outcome.
Materials and MethodsWe compared two cohort and five NCC designs for predicting early-onset gastrointestinal cancer using L1-regularized logistic regression (LASSO), XGBoost, and multilayer perceptron (MLP). Models underwent nested five-fold cross-validation with model-specific hyperparameter tuning and temporal evaluation in a held-out prospective cohort representing intended deployment. Model discrimination was assessed using area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC). Model calibration was assessed using Brier scores and calibration plots before and after recalibration with isotonic regression. Feature-ranking similarity was quantified using rank-biased overlap.
ResultsThe cohort development datasets included 256,854-415,529 patients, with 2-yr cumulative incidence of 0.12%-0.17%, compared with approximately 9% in the 1:10 matched NCC datasets. NCC designs generally produced higher internal discrimination, but temporal performance varied by algorithm and index-date definition. Cohort designs achieved the highest temporal AUPRC for LASSO (0.068), whereas diagnosis-anchored NCC achieved the highest temporal AUPRC for XGBoost (0.086) and MLP (0.070). Temporal AUPRC generally decreased as the required lead time before diagnosis increased. Raw predicted probabilities substantially overestimated observed risks in the temporal cohort. Isotonic recalibration improved probability scaling while largely preserving discrimination, although residual miscalibration remained. Cohort-based feature rankings were more consistent across algorithms, whereas NCC-based rankings depended more strongly on algorithm and index-date anchoring.
ConclusionStudy design should be selected and evaluated according to the intended clinical prediction task and deployment setting.
Related
- Dynamic Prediction to Predict Virologic Failure in Antiretroviral Treated People Living with HIV: Cohort Analysis of Routine Electronic Health Records Data in the Western Cape, South Africa
- Survival outcomes of spleen-strengthening and dampness-eliminating traditional Chinese medicine granules combined with lenvatinib for unresectable hepatocellular carcinoma: a large-sample retrospective study based on electronic medical records
- Longitudinal physical-mental multimorbidity trajectories by sociodemographic factors: a cohort study using electronic health records in Scotland (2000–2021)
- Dynamic near-term risk prediction of clinically significant immune-related adverse events using longitudinal electronic health records
- Durability of respiratory syncytial virus vaccine effectiveness for the elderly against infection and severe outcomes: A retrospective observational study with 3-year follow-up using electronic health records
BibTeX
@misc{fan2026cohort,
title = {{Cohort Versus Nested Case-Control Design for Prediction Models of Rare Outcomes Using Electronic Health Records}},
author = {Ruochong Fan and Sina Azadnajafabad and Jenna M. Reps and Benjamin Bowe and Mackenzie R. Hofford and Kian-Huat Lim and George Hripcsak and Yin Cao and Linying Zhang},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.16.26365758},
url = {https://www.medrxiv.org/content/10.64898/2026.10.16.26365758v1},
note = {CC BY-ND},
}
RIS
TY - UNPB
TI - Cohort Versus Nested Case-Control Design for Prediction Models of Rare Outcomes Using Electronic Health Records
AU - Ruochong Fan
AU - Sina Azadnajafabad
AU - Jenna M. Reps
AU - Benjamin Bowe
AU - Mackenzie R. Hofford
AU - Kian-Huat Lim
AU - George Hripcsak
AU - Yin Cao
AU - Linying Zhang
PY - 2026
DA - 2026/10/16
PB - medRxiv
DO - 10.64898/2026.10.16.26365758
UR - https://www.medrxiv.org/content/10.64898/2026.10.16.26365758v1
AB - ObjectiveTo determine how cohort versus nested case-control (NCC) design affects EHR-based prediction of a rare outcome. Materials and MethodsWe compared two cohort and five NCC designs for predicting early-onset gastrointestinal cancer using L1-regularized logistic regression (LASSO), XGBoost, and multilayer perceptron (MLP). Models underwent nested five-fold cross-validation with model-specific hyperparameter tuning and temporal evaluation in a held-out prospective cohort representing intended deployment. Model discrimination was assessed using area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC). Model calibration was assessed using Brier scores and calibration plots before and after recalibration with isotonic regression. Feature-ranking similarity was quantified using rank-biased overlap. ResultsThe cohort development datasets included 256,854-415,529 patients, with 2-yr cumulative incidence of 0.12%-0.17%, compared with approximately 9% in the 1:10 matched NCC datasets. NCC designs generally produced higher internal discrimination, but temporal performance varied by algorithm and index-date definition. Cohort designs achieved the highest temporal AUPRC for LASSO (0.068), whereas diagnosis-anchored NCC achieved the highest temporal AUPRC for XGBoost (0.086) and MLP (0.070). Temporal AUPRC generally decreased as the required lead time before diagnosis increased. Raw predicted probabilities substantially overestimated observed risks in the temporal cohort. Isotonic recalibration improved probability scaling while largely preserving discrimination, although residual miscalibration remained. Cohort-based feature rankings were more consistent across algorithms, whereas NCC-based rankings depended more strongly on algorithm and index-date anchoring. ConclusionStudy design should be selected and evaluated according to the intended clinical prediction task and deployment setting.
ER -