On the Depth Transfer of Neural Optimizers via L-Smoothness

Abstract

Scaling deep learning models efficiently requires predictable performance transfer across compute regimes. Establishing reliable hyperparameter transfer allows optimal training dynamics and structural choices to be discovered on low-cost proxy models, avoiding catastrophic failures during massive-scale training. Existing depth-transfer analyses are architecture-specific and heuristic: hyperparameter transfer is only justified through toy blocks, and both rely on infinite-depth dynamics, a specific initialization, and feature-learning criteria. Building on L-smoothness framework of Xu et al. (2026), we understand hyperparameter transfer as how do the Lipschitz and L-smoothness constants of the loss scale with the depth K? We show that inverse-depth residual branch scaling α=Θ(1/K) is sufficient to make both constants depth-independent at fixed width, uniformly over the bounded parameter domain and for arbitrary p→ q geometries. Because the bounds depend on each residual branch only through its Lipschitz and smoothness constants, the argument applies to general residual networks (including branches of block depth ≥ 2 such as transformer and ResNet blocks) without assumptions on initialization or correlations between layer weights. Since the smoothness constant controls the curvature penalty in the descent inequality, our bounds motivate a learning-rate scale without an additional depth-dependent curvature correction for updates bounded in the prescribed norm. This provides a geometric explanation of inverse-depth residual scaling and its application to the operator-norm geometries associated with AdamW, Muon, and MOGA. Learning-rate sweeps on GPT-2 (depths 2--128) and CIFAR-100 ResNets (8--32 residual blocks) confirm that Muon and MOGA exhibit stable learning-rate transfer under these prescriptions, with a common optimal rate across a 64-fold increase in depth.

Mixed-Frequency Time Series Forecasting via Depth-Separable Neural Networks
Yize Wang, Qianqian Zhu, Guodong Li
arXiv7 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Depth Separation for Neural Networks under Gaussian Measure
Cynthia Bortolotto, Francisco Ganacim, Roberto I. Oliveira
arXiv20 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Online Gate-Driven Flow Control in Resin Transfer Moulding Using a Neural-Network Surrogate
Nicholas Wright, Oliver Maclaren, Piaras Kelly +2
arXiv30 Aug 2026PreprintarXiv licenceFull textOptimisation & control StatisticsarXiv
On the Comparison of Optimizers for Imbalanced Learning
Clément Lezane, François Bachoc, Jérôme Bolte +1
HAL2 Oct 2026PreprintCC BY 4.0Full textStatisticsHALarXiv
Critical Optimizers for the Finite-Rank Fractional Lieb--Thirring Inequality
Zhuocheng Li, Xiaoyu Zeng
arXiv30 Oct 2026PreprintarXiv licenceFull textAnalysis & PDEsarXiv

Cite

BibTeX RIS

BibTeX

@misc{wong2026depth,
  title = {{On the Depth Transfer of Neural Optimizers via L-Smoothness}},
  author = {Maximilian Wong and Chenyu Xu and Jiajin Li and Yiping Lu},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.14344},
  eprint = {2610.14344},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.14344},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - On the Depth Transfer of Neural Optimizers via L-Smoothness
AU  - Maximilian Wong
AU  - Chenyu Xu
AU  - Jiajin Li
AU  - Yiping Lu
PY  - 2026
DA  - 2026/10/09
PB  - arXiv
DO  - 10.48550/arxiv.2610.14344
UR  - https://arxiv.org/abs/2610.14344
AB  - Scaling deep learning models efficiently requires predictable performance transfer across compute regimes. Establishing reliable hyperparameter transfer allows optimal training dynamics and structural choices to be discovered on low-cost proxy models, avoiding catastrophic failures during massive-scale training. Existing depth-transfer analyses are architecture-specific and heuristic: hyperparameter transfer is only justified through toy blocks, and both rely on infinite-depth dynamics, a specific initialization, and feature-learning criteria. Building on L-smoothness framework of Xu et al. (2026), we understand hyperparameter transfer as how do the Lipschitz and L-smoothness constants of the loss scale with the depth K? We show that inverse-depth residual branch scaling α=Θ(1/K) is sufficient to make both constants depth-independent at fixed width, uniformly over the bounded parameter domain and for arbitrary p→ q geometries. Because the bounds depend on each residual branch only through its Lipschitz and smoothness constants, the argument applies to general residual networks (including branches of block depth ≥ 2 such as transformer and ResNet blocks) without assumptions on initialization or correlations between layer weights. Since the smoothness constant controls the curvature penalty in the descent inequality, our bounds motivate a learning-rate scale without an additional depth-dependent curvature correction for updates bounded in the prescribed norm. This provides a geometric explanation of inverse-depth residual scaling and its application to the operator-norm geometries associated with AdamW, Muon, and MOGA. Learning-rate sweeps on GPT-2 (depths 2--128) and CIFAR-100 ResNets (8--32 residual blocks) confirm that Muon and MOGA exhibit stable learning-rate transfer under these prescriptions, with a common optimal rate across a 64-fold increase in depth.
ER  -

Abstract

Related

Cite