Finite-width shift of the edge of chaos in deep neural networks
Abstract
Deep neural networks train best when they are initialised at the edge of chaos, the critical line in the plane of weight and bias variances on which the distance between two inputs neither grows nor shrinks with depth. The line is known for infinitely wide networks; at finite width it moves, and how it moves has been disputed. We treat the logarithm of the distance between two inputs through a network of width N as a random walk and compute its drift and variance to order 1/N. The edge of chaos moves to a larger weight variance, by Δσw2=−a1/(N ∂σw2lnχ1), about 8/N for tanh networks at a bias variance of $0.05$, confirmed by simulations of the exact finite-width recursion. The shift describes a typical network; the average over many networks is dominated by rare networks in which the distance grows, and its critical point moves the other way, which explains why calculations and simulations have disagreed. Backpropagation follows the same random walk, so the shift applies to gradients unchanged. Along the edge of chaos, the information about the similarity of two inputs is constant until the compression over the depth exceeds about one nat, which fixes where to sit on the line. The results give an initialisation rule, Crinit, that places a network on its finite-width edge of chaos without a search. On MNIST and Fashion-MNIST at width $64$ it is more accurate than the standard rules at their usual settings, by a small margin in shallow networks and a large one in deep networks, where most standard rules fail; it trains in fewer steps, and it matches or beats the standard rules with a tuned gain.
Related
- The minimal width of neural networks and immersions of real projective space
- Radon Measure Representations for Infinite-Width Neural Networks with Singular Activations
- Mesoscopic learning dynamics of extensive-width neural networks
- Finite-Time Synchronization of Discontinuous Fractional-Order Multi-Layer Coupled Neural Networks via Aperiodic Intermittent Control
- Lifting Finite-Sample Gaussian Approximation of Wide Neural Networks to Functional Convergence
BibTeX
@misc{caesar2026finitewidth,
title = {{Finite-width shift of the edge of chaos in deep neural networks}},
author = {Arn Caesar},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.06582},
eprint = {2611.06582},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.06582},
}
RIS
TY - UNPB
TI - Finite-width shift of the edge of chaos in deep neural networks
AU - Arn Caesar
PY - 2026
DA - 2026/11/04
PB - arXiv
DO - 10.48550/arxiv.2611.06582
UR - https://arxiv.org/abs/2611.06582
AB - Deep neural networks train best when they are initialised at the edge of chaos, the critical line in the plane of weight and bias variances on which the distance between two inputs neither grows nor shrinks with depth. The line is known for infinitely wide networks; at finite width it moves, and how it moves has been disputed. We treat the logarithm of the distance between two inputs through a network of width N as a random walk and compute its drift and variance to order 1/N. The edge of chaos moves to a larger weight variance, by Δσ_w^2=−a_1/(N ∂_σ_w^2lnχ_1), about 8/N for tanh networks at a bias variance of $0.05$, confirmed by simulations of the exact finite-width recursion. The shift describes a typical network; the average over many networks is dominated by rare networks in which the distance grows, and its critical point moves the other way, which explains why calculations and simulations have disagreed. Backpropagation follows the same random walk, so the shift applies to gradients unchanged. Along the edge of chaos, the information about the similarity of two inputs is constant until the compression over the depth exceeds about one nat, which fixes where to sit on the line. The results give an initialisation rule, Crinit, that places a network on its finite-width edge of chaos without a search. On MNIST and Fashion-MNIST at width $64$ it is more accurate than the standard rules at their usual settings, by a small margin in shallow networks and a large one in deep networks, where most standard rules fail; it trains in fewer steps, and it matches or beats the standard rules with a tuned gain.
ER -