Design-Based Validation Method of EHR Algorithms to Address Partial Verification Bias
Abstract
Electronic health record (EHR) phenotyping algorithms support clinical prediction, observational research, and quality improvement, but manual chart review for gold-standard validation is costly. For algorithms targeting rare outcomes, algorithm-negative patients often greatly outnumber algorithm-positive patients. We propose a design-based validation framework that fully verifies algorithm-positive patients and reviews a simple random sample (SRS) of algorithm-negative patients. Horvitz-Thompson estimators of false-negative (FN) and true-negative totals yield estimates of sensitivity, specificity, negative predictive value, positive predictive value, and accuracy. We derive finite-population-corrected variance estimators and confidence intervals (CIs), with sample-size planning for a prespecified CI half-width. An optional Phase 2 extension uses pilot data to assess whether risk-guided Neyman allocation may improve precision. In simulations, the design-based approach substantially reduced bias relative to naive verified-sample analyses and achieved near-nominal CI coverage. Adaptive allocation reduced root mean squared error and CI width when FNs were concentrated across risk strata, but offered little benefit when the risk signal was weak. We illustrate the framework using a breast cancer recurrence algorithm in 3,336 patients, with endpoint-specific algorithm-negative review budgets of 125, 150, and 175 charts. The framework combines SRS-based validation and precision planning with optional pilot-guided adaptive allocation.
Related
- Conformal Prediction under Partial Verification
- Large-Scale Comparison of Metaheuristic Algorithms via Behavioral Verification of Large Language Model-Generated Codes
- Combinatorial Algorithms for Partial Inverse Min-Max Spanning Tree Problem under the Weighted Sum Hamming Distance
- On the partial convexification for low-rank spectral optimization: rank bounds and algorithms
- Robust Design and Evaluation of Predictive Algorithms under Unobserved Confounding
BibTeX
@misc{tang2026designbased,
title = {{Design-Based Validation Method of EHR Algorithms to Address Partial Verification Bias}},
author = {Rushi Tang and Maya Blasingame and Lillian Zheng and Joan Ac-Lumor and Saher Mubarek and Jennifer K. Plichta and Samuel I. Berchuck},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.12059},
eprint = {2610.12059},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.12059},
}
RIS
TY - UNPB
TI - Design-Based Validation Method of EHR Algorithms to Address Partial Verification Bias
AU - Rushi Tang
AU - Maya Blasingame
AU - Lillian Zheng
AU - Joan Ac-Lumor
AU - Saher Mubarek
AU - Jennifer K. Plichta
AU - Samuel I. Berchuck
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.12059
UR - https://arxiv.org/abs/2610.12059
AB - Electronic health record (EHR) phenotyping algorithms support clinical prediction, observational research, and quality improvement, but manual chart review for gold-standard validation is costly. For algorithms targeting rare outcomes, algorithm-negative patients often greatly outnumber algorithm-positive patients. We propose a design-based validation framework that fully verifies algorithm-positive patients and reviews a simple random sample (SRS) of algorithm-negative patients. Horvitz-Thompson estimators of false-negative (FN) and true-negative totals yield estimates of sensitivity, specificity, negative predictive value, positive predictive value, and accuracy. We derive finite-population-corrected variance estimators and confidence intervals (CIs), with sample-size planning for a prespecified CI half-width. An optional Phase 2 extension uses pilot data to assess whether risk-guided Neyman allocation may improve precision. In simulations, the design-based approach substantially reduced bias relative to naive verified-sample analyses and achieved near-nominal CI coverage. Adaptive allocation reduced root mean squared error and CI width when FNs were concentrated across risk strata, but offered little benefit when the risk signal was weak. We illustrate the framework using a breast cancer recurrence algorithm in 3,336 patients, with endpoint-specific algorithm-negative review budgets of 125, 150, and 175 charts. The framework combines SRS-based validation and precision planning with optional pilot-guided adaptive allocation.
ER -