Designing for Prediction: Planning Formulas for Expected Out-of-Sample R2 and r
Abstract
Design calculations let researchers gauge, before data is collected, whether a planned study is likely to detect an effect. In this paper, we develop such calculations for predictive models using ordinary least squares and ridge regression under leave-one-out cross-validation. To do so, we derive closed-form expressions relating the expected out-of-sample R2 and the expected correlation between observed and predicted outcomes to four design-stage quantities: sample size, number of predictors, an assumed population R2 (ρ2in) and a design efficiency factor D* (which summarizes how leverage is distributed across observations). Unlike the classical shrinkage formulas, which depend only on sample size and dimension, these are sensitive to how covariates are arranged -- to imbalance and skew. A simulation study assesses the accuracy of the formulas against realized values under a variety of data generating mechanisms. We apply the methods to a published study predicting personality from smartphone behavior and show that the design formulas would have indicated, before any analyses were run, that least squares could not be expected to beat the sample mean while ridge could. An R package, predpower, implements the calculations.
Related
- Beyond Expected Values: Risk-Sensitive Planning with Distributional Monte-Carlo Tree Search
- Expected Sample Complexity in Multi-Armed Bandits
- Calculating the Expected Value of Sample Information for Observational Studies affected by Confounding
- Metric Conformal Prediction via the Conditional Expected Local Radius
- Data-Matrix-Based Overall Goodness-of-Fit Testing for Confirmatory Factor Analysis: In-Sample Prediction Error for Observed Variables
BibTeX
@misc{tipton2026designing,
title = {{Designing for Prediction: Planning Formulas for Expected Out-of-Sample $R^2$ and $r$}},
author = {Elizabeth Tipton},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.18781},
eprint = {2610.18781},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.18781},
}
RIS
TY - UNPB
TI - Designing for Prediction: Planning Formulas for Expected Out-of-Sample R^2 and r
AU - Elizabeth Tipton
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2610.18781
UR - https://arxiv.org/abs/2610.18781
AB - Design calculations let researchers gauge, before data is collected, whether a planned study is likely to detect an effect. In this paper, we develop such calculations for predictive models using ordinary least squares and ridge regression under leave-one-out cross-validation. To do so, we derive closed-form expressions relating the expected out-of-sample R^2 and the expected correlation between observed and predicted outcomes to four design-stage quantities: sample size, number of predictors, an assumed population R^2 (ρ^2_in) and a design efficiency factor D^* (which summarizes how leverage is distributed across observations). Unlike the classical shrinkage formulas, which depend only on sample size and dimension, these are sensitive to how covariates are arranged -- to imbalance and skew. A simulation study assesses the accuracy of the formulas against realized values under a variety of data generating mechanisms. We apply the methods to a published study predicting personality from smartphone behavior and show that the design formulas would have indicated, before any analyses were run, that least squares could not be expected to beat the sample mean while ridge could. An R package, predpower, implements the calculations.
ER -