Double descent is the principle of least action
Abstract
The test error of a model plotted against its number of parameters d falls, peaks when the model can just fit the training data, and falls again, exhibiting the double descent phenomenon. We explain the phenomenon with statistical mechanics. The training trajectory of a stochastic gradient-based method is a particle wandering over the energy landscape of the training loss at an induced temperature T, and a run that has equilibrated visits every parameter vector of a given training loss equally often, the fundamental postulate of statistical mechanics, with probability given by the Boltzmann distribution. Because training starts at an initial point and has only finite time to diffuse, it carries an effective weight decay, which makes every parameter a quadratic degree of freedom. The equipartition theorem then distributes the energy among the d degrees of freedom in shares of T/2, so at a fixed training loss adding parameters lowers the temperature and drives the Boltzmann distribution toward the stationary path. Finally, adding parameters can only lower the L2 norm of the stationary path, so a solution sampled at fixed loss is less likely to be large with increasing d, effectively increasing weight regularization.
Related
- Physics of Information Geometry - Part I: Principle of Least Action on the Probability Simplex
- Randomized average block coordinate descent method with heavy-ball momentum for tensor least squares problem under the t-product
- On randomized batch-sampling coordinate descent methods for solving linear least-squares problems
- A Spectral Local-to-Global Principle for Spin Systems on Graphs with Girth At Least Five
- The double descent and Runge phenomena in overparametrized polynomial interpolation
BibTeX
@misc{sha2026double,
title = {{Double descent is the principle of least action}},
author = {Congzhou M Sha},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.19076},
eprint = {2609.19076},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.19076},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Double descent is the principle of least action
AU - Congzhou M Sha
PY - 2026
DA - 2026/09/17
PB - arXiv
DO - 10.48550/arxiv.2609.19076
UR - https://arxiv.org/abs/2609.19076
AB - The test error of a model plotted against its number of parameters d falls, peaks when the model can just fit the training data, and falls again, exhibiting the double descent phenomenon. We explain the phenomenon with statistical mechanics. The training trajectory of a stochastic gradient-based method is a particle wandering over the energy landscape of the training loss at an induced temperature T, and a run that has equilibrated visits every parameter vector of a given training loss equally often, the fundamental postulate of statistical mechanics, with probability given by the Boltzmann distribution. Because training starts at an initial point and has only finite time to diffuse, it carries an effective weight decay, which makes every parameter a quadratic degree of freedom. The equipartition theorem then distributes the energy among the d degrees of freedom in shares of T/2, so at a fixed training loss adding parameters lowers the temperature and drives the Boltzmann distribution toward the stationary path. Finally, adding parameters can only lower the L^2 norm of the stationary path, so a solution sampled at fixed loss is less likely to be large with increasing d, effectively increasing weight regularization.
ER -