Evaluating biomarkers and prediction models: predictive utility analysis with E2P Simulator
Abstract
Precision psychiatry aims to identify biomarkers and develop prediction models to improve diagnosis, prognosis, and treatment selection. Despite extensive research, translating findings into clinically useful tools remains challenging. Here we argue that one important contributor to this translational gap is a pervasive misalignment between routine statistical analytic practices and the requirements for real-world decision making. To help address this, we introduce predictive utility analysis: a statistical workflow for estimating the practical significance of biomarkers and prediction models by accounting for outcome base rates and measurement reliability. Starting from effect size or model performance estimates, the workflow simulates expected real-world performance in a target population and summarizes it across complementary metrics including positive and negative predictive value, Precision-Recall AUC, and Net Benefit. Analogous to how power analysis is used for determining the conditions needed to achieve statistical significance, predictive utility analysis can help determine the conditions needed to achieve practical significance. We implement and illustrate this workflow using E2P Simulator, a recently developed open-source web tool that makes predictive utility analysis accessible through interactive simulations and publication-ready visualizations. We demonstrate its application using specific examples in three key areas: diagnostic (depression, Alzheimer’s disease), treatment response (antidepressants), and risk (suicide attempts and psychosis onset) prediction, highlighting conditions under which we may expect clinically meaningful advances. While we focus on translational challenges in psychiatry, the workflow and tools presented here address general statistical challenges and are broadly applicable across biomedical and behavioral sciences.
Related
- Mechanistic simulation identifies predictive dose-dependent biomarkers of propofol anesthesia
- Evaluating Autonomous LLM Agents Across Molecular Prediction and Optimization Benchmarks
- Combination of iKIR missing ligand hypothesis and HLA-B T/T genotypes as predictive biomarkers for recurrent pregnancy loss: a novel algorithmic approach
- Diagnostic and Predictive CSF Biomarkers in Idiopathic Normal Pressure Hydrocephalus (iNPH): A Systematic Review
- Beyond the Final Layer: Evaluating Single-Cell Foundation Model Representations for Molecular Prediction
BibTeX
@misc{karvelis2026evaluating,
title = {{Evaluating biomarkers and prediction models: predictive utility analysis with E2P Simulator}},
author = {Povilas Karvelis and Daniel Felsky and Anissa Abi-Dargham and Guillermo Horga and Andreea Andreea},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/atn5m_v3},
url = {https://osf.io/preprints/psyarxiv/atn5m_v3/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evaluating biomarkers and prediction models: predictive utility analysis with E2P Simulator
AU - Povilas Karvelis
AU - Daniel Felsky
AU - Anissa Abi-Dargham
AU - Guillermo Horga
AU - Andreea Andreea
PY - 2026
DA - 2026/09/23
PB - PsyArXiv
DO - 10.31234/osf.io/atn5m_v3
UR - https://osf.io/preprints/psyarxiv/atn5m_v3/
AB - Precision psychiatry aims to identify biomarkers and develop prediction models to improve diagnosis, prognosis, and treatment selection. Despite extensive research, translating findings into clinically useful tools remains challenging. Here we argue that one important contributor to this translational gap is a pervasive misalignment between routine statistical analytic practices and the requirements for real-world decision making. To help address this, we introduce predictive utility analysis: a statistical workflow for estimating the practical significance of biomarkers and prediction models by accounting for outcome base rates and measurement reliability. Starting from effect size or model performance estimates, the workflow simulates expected real-world performance in a target population and summarizes it across complementary metrics including positive and negative predictive value, Precision-Recall AUC, and Net Benefit. Analogous to how power analysis is used for determining the conditions needed to achieve statistical significance, predictive utility analysis can help determine the conditions needed to achieve practical significance. We implement and illustrate this workflow using E2P Simulator, a recently developed open-source web tool that makes predictive utility analysis accessible through interactive simulations and publication-ready visualizations. We demonstrate its application using specific examples in three key areas: diagnostic (depression, Alzheimer’s disease), treatment response (antidepressants), and risk (suicide attempts and psychosis onset) prediction, highlighting conditions under which we may expect clinically meaningful advances. While we focus on translational challenges in psychiatry, the workflow and tools presented here address general statistical challenges and are broadly applicable across biomedical and behavioral sciences.
ER -