Optimal spectral shrinkage via Krylov approximation
Abstract
Spectral shrinkage encompasses a broad family of estimators for high-dimensional linear regression that regularize estimation and prediction by applying spectral filters to the sample covariance matrix. Despite its widespread use, achieving oracle-optimal prediction risk under general covariance structures remains challenging, as the oracle filter requires unknown population quantities and has a closed form only for specific covariance models such as isotropic and spiked models. We propose \textit{risk-targeted Krylov fitting} (RTK), a data-driven spectral estimator that adapts to the unknown covariance structure and attains the oracle risk asymptotically. Like partial least squares, RTK operates in an m-dimensional Krylov space, but it targets the oracle risk directly by minimizing the leave-one-out risk, which we compute exactly without refitting the model n times. Under general distributions and covariance structures, RTK attains the optimal prediction risk in the Krylov space up to O(1/n). Furthermore, its risk approaches that of the optimal analytic filter with m of order log n. In simulations over a wide range of covariance structures, RTK and a truncated variant were robust to the choice of m and consistently among the spectral estimators closest to the oracle. In applications to single-cell sequencing data and large-scale neural recordings, the proposed methods achieved higher prediction accuracy than competing estimators on most tasks.
Related
- Eigenvector distributions and optimal shrinkage estimators for large covariance and precision matrices
- Krylov geometry and optimal recovery under uncertain coherent noise
- A geometrically linear approximation of min-max optimal matching
- Super-approximation and optimal L2 convergence of the stabilized BGN method with piecewise linear parametric finite elements for curve-shortening flow
- Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexity
BibTeX
@misc{lu2026optimal,
title = {{Optimal spectral shrinkage via Krylov approximation}},
author = {Tianfeng Lu and Geoffrey J. Goodhill and Hong Hu},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.02122},
eprint = {2611.02122},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.02122},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Optimal spectral shrinkage via Krylov approximation
AU - Tianfeng Lu
AU - Geoffrey J. Goodhill
AU - Hong Hu
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.02122
UR - https://arxiv.org/abs/2611.02122
AB - Spectral shrinkage encompasses a broad family of estimators for high-dimensional linear regression that regularize estimation and prediction by applying spectral filters to the sample covariance matrix. Despite its widespread use, achieving oracle-optimal prediction risk under general covariance structures remains challenging, as the oracle filter requires unknown population quantities and has a closed form only for specific covariance models such as isotropic and spiked models. We propose \textit{risk-targeted Krylov fitting} (RTK), a data-driven spectral estimator that adapts to the unknown covariance structure and attains the oracle risk asymptotically. Like partial least squares, RTK operates in an m-dimensional Krylov space, but it targets the oracle risk directly by minimizing the leave-one-out risk, which we compute exactly without refitting the model n times. Under general distributions and covariance structures, RTK attains the optimal prediction risk in the Krylov space up to O(1/n). Furthermore, its risk approaches that of the optimal analytic filter with m of order log n. In simulations over a wide range of covariance structures, RTK and a truncated variant were robust to the choice of m and consistently among the spectral estimators closest to the oracle. In applications to single-cell sequencing data and large-scale neural recordings, the proposed methods achieved higher prediction accuracy than competing estimators on most tasks.
ER -