Large deviations for linear regressions
Abstract
Linear regression is one of the simplest and most widely used tools to learn patterns from data: it fits a set of coefficients so that a linear combination of predictors best matches observed responses. The quality of the fit is measured by the residual sum of squares, the total squared mismatch between predictions and data, whose minimum defines the training loss. We consider Gaussian design and noise, with teacher coefficients independently drawn from a general distribution p(β), and a general class of separable regularizers, including Ridge and Lasso. Using the zero-temperature replica method, we compute analytically the large-deviation statistics of the minimum training loss for large numbers P of predictors and N of observations, with r=P/N fixed. The rate function we compute governs rare sample-to-sample fluctuations of the optimal loss. Extensive numerical simulations are in excellent agreement with our theory and clearly show a pronounced deviation from the Gaussian regime of typical fluctuations in the tails.
Related
- Dynamical large deviations, self-similarity, and criticality
- Large deviations in the many-body localization transition: The case of the random-field XXZ chain
- Large Deviations for the Bose--Einstein Condensate of the Ideal Gas in the Canonical Ensemble
- Large deviations, gradient flows and fluctuating hydrodynamics for fractional diffusion
- Spectral theory for dynamical large deviations in non-Markov self-interacting processes
BibTeX
@misc{bartolucci2026large,
title = {{Large deviations for linear regressions}},
author = {Silvia Bartolucci and Fabio Caccioli and Francesco Caravelli and Pierpaolo Vivo},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.03111},
eprint = {2609.03111},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.03111},
}
RIS
TY - UNPB
TI - Large deviations for linear regressions
AU - Silvia Bartolucci
AU - Fabio Caccioli
AU - Francesco Caravelli
AU - Pierpaolo Vivo
PY - 2026
DA - 2026/09/02
PB - arXiv
DO - 10.48550/arxiv.2609.03111
UR - https://arxiv.org/abs/2609.03111
AB - Linear regression is one of the simplest and most widely used tools to learn patterns from data: it fits a set of coefficients so that a linear combination of predictors best matches observed responses. The quality of the fit is measured by the residual sum of squares, the total squared mismatch between predictions and data, whose minimum defines the training loss. We consider Gaussian design and noise, with teacher coefficients independently drawn from a general distribution p(β), and a general class of separable regularizers, including Ridge and Lasso. Using the zero-temperature replica method, we compute analytically the large-deviation statistics of the minimum training loss for large numbers P of predictors and N of observations, with r=P/N fixed. The rate function we compute governs rare sample-to-sample fluctuations of the optimal loss. Extensive numerical simulations are in excellent agreement with our theory and clearly show a pronounced deviation from the Gaussian regime of typical fluctuations in the tails.
ER -