Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity
Abstract
Deep learning models tend to perform best at large widths and depths. The growing interest in scaling laws, especially in the language modeling domain, has motivated procedures for scaling up network widths and depths in order to preserve stability and computational efficiency of training dynamics. Existing depth-scaling approaches force a choice between retaining locally maximal \emph{feature learning} and preserving a form of \emph{feature diversity} in the residual stream arising from random differences between channels. We examine a \textit{joint scaling recipe} that preserves both. With residual width N♭, hidden width N♯, and depth L, the key quantity is, by default, the limiting ratio N♭/(N♯ L)→α, while correlated initialization (Half Copy) can partially relax the resulting shape constraints. Theory and residual-MLP experiments support convergence at fixed positive α to an SDE limit with locally maximal feature learning. Hidden computations can then be redistributed between width and depth while preserving this limit, like turning melody into harmony and back. Language-model experiments nevertheless show performance differences under regrouping at fixed parameter count. These results distinguish approximating a limit (ODE or SDE) from scaling for better performance.
Related
- Early learning in large residual networks
- TCSP3.0: Template-Based Crystal Structure Prediction with Coefficient-Preserving Element Substitution and Structural Diversity
- Many-body topology in parity-preserving tensor networks
- Mesoscale structures in signed networks
- Label-Free Finite-Volume-Residual Training of Attention Graph Neural Networks for Coupled Thermo-Fluid Fields
BibTeX
@misc{zhou2026sliding,
title = {{Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity}},
author = {Datong Zhou and Blake Bordelon},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.33840},
eprint = {2610.33840},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33840},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity
AU - Datong Zhou
AU - Blake Bordelon
PY - 2026
DA - 2026/10/24
PB - arXiv
DO - 10.48550/arxiv.2610.33840
UR - https://arxiv.org/abs/2610.33840
AB - Deep learning models tend to perform best at large widths and depths. The growing interest in scaling laws, especially in the language modeling domain, has motivated procedures for scaling up network widths and depths in order to preserve stability and computational efficiency of training dynamics. Existing depth-scaling approaches force a choice between retaining locally maximal \emph{feature learning} and preserving a form of \emph{feature diversity} in the residual stream arising from random differences between channels. We examine a \textit{joint scaling recipe} that preserves both. With residual width N_♭, hidden width N_♯, and depth L, the key quantity is, by default, the limiting ratio N_♭/(N_♯ L)→α, while correlated initialization (Half Copy) can partially relax the resulting shape constraints. Theory and residual-MLP experiments support convergence at fixed positive α to an SDE limit with locally maximal feature learning. Hidden computations can then be redistributed between width and depth while preserving this limit, like turning melody into harmony and back. Language-model experiments nevertheless show performance differences under regrouping at fixed parameter count. These results distinguish approximating a limit (ODE or SDE) from scaling for better performance.
ER -