Max-Test-Calibrated Stein Shrinkage with Honest Submodel Selection in Ultra-High-Dimensional Regression
Abstract
Classical preliminary-test and Stein-type estimators interpolate between restricted and full regression fits, but OLS and chi-squared calibration fail when p≫ n and the restriction is data-adaptive. We propose an honest sample-separated framework built around a common selected-null law. Independent selection data extend a mandatory core; separate estimation data pair a complete-design regularized full-model (FM) fit with a fresh exact-null submodel refit. A maximum residual-association statistic assesses all excluded coordinates. Projected Gaussian draws reproduce its conditional null distribution under homoskedastic Gaussian errors, yielding a finite-sample-valid rank test. The inverse moment of the same design-specific law replaces q−2 and calibrates preliminary-test (PT), Stein-type (S), and positive-part Stein-type (PS) rules. We derive exact endpoint-risk formulas, honest conditional validity, null-law concentration, inverse-moment consistency, an explicit selection-failure remainder, and endpoint adaptivity without uniform-dominance claims. Gaussian experiments with 2,000 replications and up to 30,000 predictors compare Ridge and LASSO FMs, tuned by frozen-fold cross-validation, under a common submodel, test, and calibration. PS provides large near-null coefficient-risk gains and approaches the relevant full-model risk under strong departures. A separate CPSS-LASSO/CPSS-MCP audit evaluates data-adaptive selection, while a split-sample DepMap study with 19,152 predictors illustrates prediction gains and sensitivity to error assumptions. The HDMaxShrink R package implements the procedure.
Related
- The Lambert Penalty: Logarithmic Shrinkage for Sparse Regression
- Shrinkage invalidates the Hosmer-Lemeshow test: goodness of fit for penalized logistic regression, with an application to glaucoma diagnosis
- Nonparametric shrinkage estimation of high-dimensional variance component matrices
- Heterogeneity-calibrated Byzantine-robust distributed composite quantile regression
- Calibrated Bayesian Selection of Schema Subgraphs for Database Question Answering
BibTeX
@misc{yuzbası2026maxtestcalibrated,
title = {{Max-Test-Calibrated Stein Shrinkage with Honest Submodel Selection in Ultra-High-Dimensional Regression}},
author = {Bahadır Yüzbaşı},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.23070},
eprint = {2609.23070},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.23070},
}
RIS
TY - UNPB
TI - Max-Test-Calibrated Stein Shrinkage with Honest Submodel Selection in Ultra-High-Dimensional Regression
AU - Bahadır Yüzbaşı
PY - 2026
DA - 2026/09/19
PB - arXiv
DO - 10.48550/arxiv.2609.23070
UR - https://arxiv.org/abs/2609.23070
AB - Classical preliminary-test and Stein-type estimators interpolate between restricted and full regression fits, but OLS and chi-squared calibration fail when p≫ n and the restriction is data-adaptive. We propose an honest sample-separated framework built around a common selected-null law. Independent selection data extend a mandatory core; separate estimation data pair a complete-design regularized full-model (FM) fit with a fresh exact-null submodel refit. A maximum residual-association statistic assesses all excluded coordinates. Projected Gaussian draws reproduce its conditional null distribution under homoskedastic Gaussian errors, yielding a finite-sample-valid rank test. The inverse moment of the same design-specific law replaces q−2 and calibrates preliminary-test (PT), Stein-type (S), and positive-part Stein-type (PS) rules. We derive exact endpoint-risk formulas, honest conditional validity, null-law concentration, inverse-moment consistency, an explicit selection-failure remainder, and endpoint adaptivity without uniform-dominance claims. Gaussian experiments with 2,000 replications and up to 30,000 predictors compare Ridge and LASSO FMs, tuned by frozen-fold cross-validation, under a common submodel, test, and calibration. PS provides large near-null coefficient-risk gains and approaches the relevant full-model risk under strong departures. A separate CPSS-LASSO/CPSS-MCP audit evaluates data-adaptive selection, while a split-sample DepMap study with 19,152 predictors illustrates prediction gains and sensitivity to error assumptions. The HDMaxShrink R package implements the procedure.
ER -