Width and Depth Limits Commute During Training in Linear Residual Networks

Abstract

Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by 1/√(depth). In particular, the network output converges to the same limit whether width and depth tend to infinity sequentially or jointly. We extend this commutativity result to training in linear residual networks. Under the same 1/√(depth) residual-branch scaling (Depth-μP) and the μP width parameterization, we prove that, for any fixed number of gradient-descent steps with gradient clipping, the network output converges in L² along every joint width--depth growth path as both width and depth tend to infinity. Consequently, the infinite-width and infinite-depth limits commute throughout any finite training horizon. Without gradient clipping, the corresponding L² convergence does not hold in general, but we establish the weaker convergence in probability.

A Two-Residual Compactness Framework for Suitable Navier–Stokes Limits
Suares Clovis Oukouomi Noutchie
Mathematics19 Oct 2026Peer-reviewedCC BY 4.0Full textGeneral mathematicsCrossrefDOAJ
Polynomial Residual Networks are Generically Identifiable
Isabel Dahlgren, Giovanni Luca Marchetti, Kathlén Kohn
arXiv24 Oct 2026PreprintCC BY 4.0Full textAlgebra & number theoryarXiv
Approximation of solutions of parameter-dependent problems by residual neural networks
Ana Carpio
arXiv15 Jul 2026PreprintarXiv licenceFull textAnalysis & PDEs Numerical analysis & scientific computingarXiv
LAIR-Net: Leaky Alignment-Impulse Residual Networks for Tabular Regression
Rahul Goswami, Aryan Bhambu, Bittu Karmakar
arXiv8 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Adaptive Randomized Neural Networks for PDEs: Structure-Aware Feature Redistribution and Residual-Driven Block Enrichment
You Yang, Fei Wang
arXiv15 Oct 2026PreprintarXiv licenceFull textNumerical analysis & scientific computingarXiv

Cite

BibTeX RIS

BibTeX

@misc{zhang2026width,
  title = {{Width and Depth Limits Commute During Training in Linear Residual Networks}},
  author = {Ruijia Zhang and Soufiane Hayou},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.30729},
  eprint = {2610.30729},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.30729},
}

RIS

TY  - UNPB
TI  - Width and Depth Limits Commute During Training in Linear Residual Networks
AU  - Ruijia Zhang
AU  - Soufiane Hayou
PY  - 2026
DA  - 2026/10/26
PB  - arXiv
DO  - 10.48550/arxiv.2610.30729
UR  - https://arxiv.org/abs/2610.30729
AB  - Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by 1/√(depth). In particular, the network output converges to the same limit whether width and depth tend to infinity sequentially or jointly. We extend this commutativity result to training in linear residual networks. Under the same 1/√(depth) residual-branch scaling (Depth-μP) and the μP width parameterization, we prove that, for any fixed number of gradient-descent steps with gradient clipping, the network output converges in L^2 along every joint width--depth growth path as both width and depth tend to infinity. Consequently, the infinite-width and infinite-depth limits commute throughout any finite training horizon. Without gradient clipping, the corresponding L^2 convergence does not hold in general, but we establish the weaker convergence in probability.
ER  -

Abstract

Related

Cite