Nonparametric shrinkage estimation of high-dimensional variance component matrices
Abstract
We develop a nonparametric spectral estimation method for variance component covariance matrices in high-dimensional multivariate mixed effects models. Although classical MANOVA estimators are unbiased entrywise, their eigenvalues can be severely dispersed and biased when the number of traits is comparable to the number of realizations of the random effects. Building on a system of self-consistent equations that characterizes the limiting spectral distribution of a MANOVA estimator, we construct a kernel-based nonlinear transformation of the sample eigenvalues that targets the optimal rotation-invariant estimator under Frobenius loss. The resulting procedure corrects for high-dimensional eigenvalue spreading without imposing a parametric model on the unknown covariance spectrum. Our work is motivated by applications in quantitative genetics, where estimating genetic covariance components across many phenotypic traits is of central interest. We implement the method in a new Python package mlmm-shrinkage and compare the proposed estimator with classical estimators across several common experimental designs in this domain through simulations and a real-data application.
Related
- Improved Variance Estimation in Homoskedastic Nonparametric Random-Design Regression via a Two-Scale Approach
- Optimal Neural Estimation for High-Dimensional Nonparametric Regression under Multisource Covariate Shift
- Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
- Nonparametric hazard rate estimation with associated kernels and minimax bandwidth choice
- Tuning-Free Covariance-Aware Sequential Shrinkage for Multi-Source Estimation
BibTeX
@misc{xie2026nonparametric,
title = {{Nonparametric shrinkage estimation of high-dimensional variance component matrices}},
author = {Ran Xie and Iain M. Johnstone},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.33431},
eprint = {2610.33431},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33431},
}
RIS
TY - UNPB
TI - Nonparametric shrinkage estimation of high-dimensional variance component matrices
AU - Ran Xie
AU - Iain M. Johnstone
PY - 2026
DA - 2026/10/24
PB - arXiv
DO - 10.48550/arxiv.2610.33431
UR - https://arxiv.org/abs/2610.33431
AB - We develop a nonparametric spectral estimation method for variance component covariance matrices in high-dimensional multivariate mixed effects models. Although classical MANOVA estimators are unbiased entrywise, their eigenvalues can be severely dispersed and biased when the number of traits is comparable to the number of realizations of the random effects. Building on a system of self-consistent equations that characterizes the limiting spectral distribution of a MANOVA estimator, we construct a kernel-based nonlinear transformation of the sample eigenvalues that targets the optimal rotation-invariant estimator under Frobenius loss. The resulting procedure corrects for high-dimensional eigenvalue spreading without imposing a parametric model on the unknown covariance spectrum. Our work is motivated by applications in quantitative genetics, where estimating genetic covariance components across many phenotypic traits is of central interest. We implement the method in a new Python package mlmm-shrinkage and compare the proposed estimator with classical estimators across several common experimental designs in this domain through simulations and a real-data application.
ER -