A prism hierarchy of learning regimes in large linear autoencoders
Abstract
Theoretical studies of machine learning models commonly consider different limiting regimes in which the learning dynamics of gradient descent becomes theoretically tractable. It is, however, desirable to have a systematically obtained picture of qualitatively different extreme learning regimes for a particular type of models. In this paper we propose such a picture for large weight-tied linear autoencoders characterized by input and latent dimensions, initialization magnitude, and training set size. This model is nonlinear in the weights and its gradient flow does not have a general theoretical solution. We show that at the level of the formal loss-expansion hierarchy, its extreme regimes are naturally associated with faces of a triangular prism. In particular, there are five basic extreme regimes associated with the 2-faces of the prism: (1) large-data, (2) small-data, (3) mean-field, (4) narrow-latent, and (5) free. For regimes (1,2,3,4), we derive and rigorously characterize the limiting train and population dynamics under gradient flow, rigorously prove convergence to these limits, and get good agreement with experimental results. The resulting limits reveal spectral critical slowing, initialization-driven trapping away from the PCA optimum, and regimes in which population loss converges exponentially while training loss relaxes only algebraically.
Related
- Sparse Koopman Autoencoders Identify Local Dynamical Regimes in Multibasin Systems
- Broken scale symmetries in undercomplete linear autoencoders
- Functional Autoencoders for Amplitude-Phase Representation Learning
- Autoencoders vs. Numerical Analysis--Informed Manifold Learning for Navier--Stokes Flows
- Bilevel optimization for data-driven learning of Koopman embeddings using kernel-based autoencoders
BibTeX
@misc{golikov2026prism,
title = {{A prism hierarchy of learning regimes in large linear autoencoders}},
author = {Eugene Golikov and Yaroslav Gusev and Dmitry Yarotsky},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2606.05335},
eprint = {2606.05335},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.05335},
}
RIS
TY - UNPB
TI - A prism hierarchy of learning regimes in large linear autoencoders
AU - Eugene Golikov
AU - Yaroslav Gusev
AU - Dmitry Yarotsky
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2606.05335
UR - https://arxiv.org/abs/2606.05335
AB - Theoretical studies of machine learning models commonly consider different limiting regimes in which the learning dynamics of gradient descent becomes theoretically tractable. It is, however, desirable to have a systematically obtained picture of qualitatively different extreme learning regimes for a particular type of models. In this paper we propose such a picture for large weight-tied linear autoencoders characterized by input and latent dimensions, initialization magnitude, and training set size. This model is nonlinear in the weights and its gradient flow does not have a general theoretical solution. We show that at the level of the formal loss-expansion hierarchy, its extreme regimes are naturally associated with faces of a triangular prism. In particular, there are five basic extreme regimes associated with the 2-faces of the prism: (1) large-data, (2) small-data, (3) mean-field, (4) narrow-latent, and (5) free. For regimes (1,2,3,4), we derive and rigorously characterize the limiting train and population dynamics under gradient flow, rigorously prove convergence to these limits, and get good agreement with experimental results. The resulting limits reveal spectral critical slowing, initialization-driven trapping away from the PCA optimum, and regimes in which population loss converges exponentially while training loss relaxes only algebraically.
ER -