Tuning-Free Covariance-Aware Sequential Shrinkage for Multi-Source Estimation
Abstract
Modern statistical learning problems often involve multiple related data sets, in which learning efficiency on a target set can be improved by borrowing information from related sources, while source heterogeneity may introduce bias. Existing approaches are limited by suboptimal performance in multi-source settings, insufficient use of covariance information, or the computational burden of tuning procedures. We propose a covariance-aware shrinkage framework that constructs shrinkage directions using covariance information to improve efficiency. We establish finite-sample risk bounds that yield an explicit risk-improving interval for the shrinkage size, making the transfer strength fully data-driven. When multiple source sets are available, we further propose a sequential algorithm that applies source-specific shrinkage updates according to a priority score. Under a mixed-source separation condition, the estimator asymptotically attains the oracle risk and, when the pooled discrepancy is large, strictly improves over single-step shrinkage. The framework is further extended to a broad class of smooth M-estimation problems. Numerical studies show substantial gains over competing methods, especially when the source data sets are highly heterogeneous.
Related
- Eigenvector distributions and optimal shrinkage estimators for large covariance and precision matrices
- Nonparametric shrinkage estimation of high-dimensional variance component matrices
- Shrinkage Estimation for Spherically Symmetric Distributions under Modified Balanced Loss Functions
- Robust Covariance Estimation for Entry-wise Contaminated and Partially Observed Data via Expectation Maximization Algorithm
- Online Covariance Estimation for Averaged Stochastic Gradient Descent on Riemannian Manifolds
BibTeX
@misc{jing2026tuningfree,
title = {{Tuning-Free Covariance-Aware Sequential Shrinkage for Multi-Source Estimation}},
author = {Wenbo Jing and Xi Chen and Yaqi Duan and Kaizheng Wang and Yichen Zhang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.30615},
eprint = {2606.30615},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.30615},
}
RIS
TY - UNPB
TI - Tuning-Free Covariance-Aware Sequential Shrinkage for Multi-Source Estimation
AU - Wenbo Jing
AU - Xi Chen
AU - Yaqi Duan
AU - Kaizheng Wang
AU - Yichen Zhang
PY - 2026
DA - 2026/09/21
PB - arXiv
DO - 10.48550/arxiv.2606.30615
UR - https://arxiv.org/abs/2606.30615
AB - Modern statistical learning problems often involve multiple related data sets, in which learning efficiency on a target set can be improved by borrowing information from related sources, while source heterogeneity may introduce bias. Existing approaches are limited by suboptimal performance in multi-source settings, insufficient use of covariance information, or the computational burden of tuning procedures. We propose a covariance-aware shrinkage framework that constructs shrinkage directions using covariance information to improve efficiency. We establish finite-sample risk bounds that yield an explicit risk-improving interval for the shrinkage size, making the transfer strength fully data-driven. When multiple source sets are available, we further propose a sequential algorithm that applies source-specific shrinkage updates according to a priority score. Under a mixed-source separation condition, the estimator asymptotically attains the oracle risk and, when the pooled discrepancy is large, strictly improves over single-step shrinkage. The framework is further extended to a broad class of smooth M-estimation problems. Numerical studies show substantial gains over competing methods, especially when the source data sets are highly heterogeneous.
ER -