Correlation flow governs learning at criticality
Abstract
The initialization of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive. Combining mean-field theory and random matrix theory, we establish a direct link between correlation propagation and the Neural Tangent Kernel (NTK) that governs learning in the sequential limit of infinitely wide, infinitely deep networks. Correlation propagation to infinite depth is possible only at a single, critical point in the weight-bias variance plane. At this point, we leverage the algebraic decay of the end-to-end Jacobian with depth to prove that the NTK becomes exactly proportional to the output correlation at infinite depth, tying together information propagation and learning dynamics. We further show that orthogonal initialization suppresses the leading finite-size corrections present under Gaussian initialization, clarifying the respective roles of the two initialization ensembles in this limit. These theoretical predictions are validated quantitatively on finite-width, finite-depth networks. Together, these results demonstrate that orthogonal initialization and criticality are required to control the asymptotic dynamics of deep learning.
Related
- When irrelevance governs: dissipation-regression scaling in open quantum criticality
- Cluster Representation of Renormalization Group Transformations and a Rigorous Proof for Convergence of the RG-Flow of the Ising Model to Trivial Fixed Points away from Criticality
- Landau theory of quenched criticality in linear in-context learning
- Compression of turbulent flow data through an extended space-time correlation: certification against the modal energy budget
- Extracting target-related flow structures using a time-lagged cross-correlation ranking criterion
BibTeX
@misc{combette2026correlation,
title = {{Correlation flow governs learning at criticality}},
author = {Andrea Combette and Nelly Pustelnik and Antoine Venaille},
howpublished = {HAL},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.08350},
eprint = {2608.08350},
archivePrefix = {arXiv},
url = {https://hal.science/hal-05761710v1},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Correlation flow governs learning at criticality
AU - Andrea Combette
AU - Nelly Pustelnik
AU - Antoine Venaille
PY - 2026
DA - 2026/08/31
PB - HAL
DO - 10.48550/arxiv.2608.08350
UR - https://hal.science/hal-05761710v1
AB - The initialization of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive. Combining mean-field theory and random matrix theory, we establish a direct link between correlation propagation and the Neural Tangent Kernel (NTK) that governs learning in the sequential limit of infinitely wide, infinitely deep networks. Correlation propagation to infinite depth is possible only at a single, critical point in the weight-bias variance plane. At this point, we leverage the algebraic decay of the end-to-end Jacobian with depth to prove that the NTK becomes exactly proportional to the output correlation at infinite depth, tying together information propagation and learning dynamics. We further show that orthogonal initialization suppresses the leading finite-size corrections present under Gaussian initialization, clarifying the respective roles of the two initialization ensembles in this limit. These theoretical predictions are validated quantitatively on finite-width, finite-depth networks. Together, these results demonstrate that orthogonal initialization and criticality are required to control the asymptotic dynamics of deep learning.
ER -