Why Heavy-Tailed Weight Spectra Predict Model Quality
Abstract
A solid understanding of the predictive success of deep neural networks (DNNs) remains elusive. For example, while many data-dependent metrics exist that seek to predict model quality of DNNs, these metrics may be expensive to compute for large datasets and/or may be impossible to compute when training datasets are not released. A practical and effective predictor of model quality involves analyzing DNN weight matrices, as prior work has established that heavy-tailed spectral distributions in trained weight matrices can be strongly predictive of model quality. However, such metrics lack a rigorous statistical justification. In this work, we aim to understand why these metrics perform well, from a statistical perspective. To do so, we employ a probabilistic framework to derive the marginal likelihood for a DNN, evaluated using only the final layer weights. Across large-scale convolutional DNNs and LLMs, we find that this marginal likelihood predicts model quality. To understand the role heavy-tailed spectra play in predictive performance, we derive the limiting log-marginal likelihood for spectra that follow the (heavy-tailed) High-Temperature Marchenko-Pastur (HTMP) distribution. We show that this limiting quantity is convex in a heaviness parameter, and we derive an optimal heaviness that empirically predicts over-fitting for large-scale DNNs.
Related
- Limit theory for a class of non-stationary linear processes with weakly dependent heavy-tailed innovations
- Tight Heavy-Tailed Complexity Bounds for Optimizing Polyak--Łojasiewicz Objectives
- DynaSPLIT: Strong tail-optimal scheduling for the M/G/n with heavy-tailed job sizes
- Hypocoercivity of Tempered Bouncy Particle Samplers for Heavy-Tailed Targets
- Spectral aspects of random heavy-tailed tensors
BibTeX
@misc{wilson2026heavytailed,
title = {{Why Heavy-Tailed Weight Spectra Predict Model Quality}},
author = {Joseph Wilson and Chris van der Heide and Liam Hodgkinson and Zhichao Wang and Fred Roosta and Michael W. Mahoney},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.13123},
eprint = {2610.13123},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.13123},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Why Heavy-Tailed Weight Spectra Predict Model Quality
AU - Joseph Wilson
AU - Chris van der Heide
AU - Liam Hodgkinson
AU - Zhichao Wang
AU - Fred Roosta
AU - Michael W. Mahoney
PY - 2026
DA - 2026/10/09
PB - arXiv
DO - 10.48550/arxiv.2610.13123
UR - https://arxiv.org/abs/2610.13123
AB - A solid understanding of the predictive success of deep neural networks (DNNs) remains elusive. For example, while many data-dependent metrics exist that seek to predict model quality of DNNs, these metrics may be expensive to compute for large datasets and/or may be impossible to compute when training datasets are not released. A practical and effective predictor of model quality involves analyzing DNN weight matrices, as prior work has established that heavy-tailed spectral distributions in trained weight matrices can be strongly predictive of model quality. However, such metrics lack a rigorous statistical justification. In this work, we aim to understand why these metrics perform well, from a statistical perspective. To do so, we employ a probabilistic framework to derive the marginal likelihood for a DNN, evaluated using only the final layer weights. Across large-scale convolutional DNNs and LLMs, we find that this marginal likelihood predicts model quality. To understand the role heavy-tailed spectra play in predictive performance, we derive the limiting log-marginal likelihood for spectra that follow the (heavy-tailed) High-Temperature Marchenko-Pastur (HTMP) distribution. We show that this limiting quantity is convex in a heaviness parameter, and we derive an optimal heaviness that empirically predicts over-fitting for large-scale DNNs.
ER -