Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective
Abstract
We study gradient flow in a two-layer linear network in the high-dimensional regime where the input, inner and output dimensions diverge proportionally. Under whitened inputs and isotropic imbalance, we combine a closed-form solution of the dynamics with prior random matrix theory results to derive a high-dimensional limit for the dynamics. Our method yields exact asymptotic spectral distributions for the weight matrices at all times and characterizes the evolution of the learned subspace. In the small-initialization limit, we identify a silent alignment phenomenon: the singular directions of the network align on an order-one timescale, while the singular values and empirical loss remain at their initial values. Visible learning occurs later, on a logarithmic timescale, as singular values progressively escape zero and the loss decreases. This phase is characterized by a saddle-to-saddle dynamics, which becomes a continuous process of spectral learning in the high-dimensional limit.
Related
- A linear dual-field finite element method for incompressible MHD, part I: cross-helicity and Alfvénic alignment in periodic domains
- LAIR-Net: Leaky Alignment-Impulse Residual Networks for Tabular Regression
- A new perspective in linear Cauchy Elasticity: variational minimum principles for statics, dynamics, and heterogeneous materials
- Optimal Initialization Scale for Neural Networks with Locally Quadratic Loss Landscapes: An SGD Dynamics Perspective
- Exact Fragmentation and Terminal-Cluster Selection in One-Dimensional Finite-Range Normalized Alignment
BibTeX
@misc{martin2026silent,
title = {{Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective}},
author = {Simon Martin and Francis Bach and Giulio Biroli},
howpublished = {HAL},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.29864},
eprint = {2610.29864},
archivePrefix = {arXiv},
url = {https://hal.science/hal-05824844v1},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective
AU - Simon Martin
AU - Francis Bach
AU - Giulio Biroli
PY - 2026
DA - 2026/10/20
PB - HAL
DO - 10.48550/arxiv.2610.29864
UR - https://hal.science/hal-05824844v1
AB - We study gradient flow in a two-layer linear network in the high-dimensional regime where the input, inner and output dimensions diverge proportionally. Under whitened inputs and isotropic imbalance, we combine a closed-form solution of the dynamics with prior random matrix theory results to derive a high-dimensional limit for the dynamics. Our method yields exact asymptotic spectral distributions for the weight matrices at all times and characterizes the evolution of the learned subspace. In the small-initialization limit, we identify a silent alignment phenomenon: the singular directions of the network align on an order-one timescale, while the singular values and empirical loss remain at their initial values. Visible learning occurs later, on a logarithmic timescale, as singular values progressively escape zero and the loss decreases. This phase is characterized by a saddle-to-saddle dynamics, which becomes a continuous process of spectral learning in the high-dimensional limit.
ER -