Self-Distillation is Optimal Among Spectral Shrinkage Estimators
Abstract
Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in high-dimensional linear regression by introducing and analyzing a broad class of estimators, namely spectral shrinkage estimators. We consider both spiked and general covariance matrices. For the former, we establish that when the covariance has s distinct spikes, s-step self-distillation achieves optimal prediction among spectral shrinkage estimators, outperforming well-known estimators in statistics and machine learning. Moreover, we show that s steps are necessary for optimality: any (s−k)-step distilled estimator is strictly suboptimal for 1 ≤ k ≤ s. For general covariance matrices, we show that self-distillation approaches the optimal limiting prediction risk among continuous spectral shrinkage estimators as the number of distillation steps increases. We further develop a generalized cross-validation procedure for selecting the distillation parameters from data and establish an asymptotic oracle guarantee for the resulting estimator. We also study a federated approach where multiple data centers share spectral shrinkage estimators and a common server seeks to aggregate them to achieve optimal performance. In this case, we find that the best local rule again takes the form of self-distillation, though it differs from the optimal rule when data are hosted centrally on a single server. Together, our results explain why self-distillation improves predictive performance and provide a broader statistical framework that connects it to classical shrinkage-based methods.
Related
- Eigenvector distributions and optimal shrinkage estimators for large covariance and precision matrices
- Optimal spectral shrinkage via Krylov approximation
- Optimal Two‐Phase Sampling Designs for Generalized Raking Estimators With Multiple Parameters of Interest
- Birthday Estimators Are Uniformly Optimal for Renyi Entropy
- Minimax-Optimal Static Estimators Under Interference
BibTeX
@misc{lecoiu2026selfdistillation,
title = {{Self-Distillation is Optimal Among Spectral Shrinkage Estimators}},
author = {Radu Lecoiu and Debarghya Mukherjee and Pragya Sur},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2605.17778},
eprint = {2605.17778},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.17778},
}
RIS
TY - UNPB
TI - Self-Distillation is Optimal Among Spectral Shrinkage Estimators
AU - Radu Lecoiu
AU - Debarghya Mukherjee
AU - Pragya Sur
PY - 2026
DA - 2026/11/05
PB - arXiv
DO - 10.48550/arxiv.2605.17778
UR - https://arxiv.org/abs/2605.17778
AB - Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in high-dimensional linear regression by introducing and analyzing a broad class of estimators, namely spectral shrinkage estimators. We consider both spiked and general covariance matrices. For the former, we establish that when the covariance has s distinct spikes, s-step self-distillation achieves optimal prediction among spectral shrinkage estimators, outperforming well-known estimators in statistics and machine learning. Moreover, we show that s steps are necessary for optimality: any (s−k)-step distilled estimator is strictly suboptimal for 1 ≤ k ≤ s. For general covariance matrices, we show that self-distillation approaches the optimal limiting prediction risk among continuous spectral shrinkage estimators as the number of distillation steps increases. We further develop a generalized cross-validation procedure for selecting the distillation parameters from data and establish an asymptotic oracle guarantee for the resulting estimator. We also study a federated approach where multiple data centers share spectral shrinkage estimators and a common server seeks to aggregate them to achieve optimal performance. In this case, we find that the best local rule again takes the form of self-distillation, though it differs from the optimal rule when data are hosted centrally on a single server. Together, our results explain why self-distillation improves predictive performance and provide a broader statistical framework that connects it to classical shrinkage-based methods.
ER -