Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks
Abstract
Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the exponents quantifying how performance changes with resources. Recent theory has shown how scaling exponents are shaped by the data distribution and the structure of the learning task, but whether network depth can improve training-time scaling has remained unsettled. Through a refined analysis of deep linear networks, we clarify that the task spectrum determines when increasing network depth improves training-time scaling. For spectrally difficult tasks, increasing depth improves the asymptotic training-time scaling exponent, whereas for easy tasks, the exponent remains independent of depth, with depth affecting only a logarithmic correction at the boundary between these regimes. Remarkably, we find that a reparameterization of time governed by the predictor norm provides a unified description of the dynamics across task difficulties. Under this reparameterization, the loss follows a common scaling law whose exponent is independent of depth. Numerical experiments on a task derived from benchmark image data and with nonlinear networks exhibit qualitatively consistent task-dependent effects of depth on training-time scaling. Our findings advance the theoretical understanding of how task structure shapes the effects of network architecture on learning and offer a basis for studying scaling behavior across broader classes of neural networks.
Related
- Width and Depth Limits Commute During Training in Linear Residual Networks
- Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons
- Impact of Feature Scaling on the Convergence Speed of Linear SVM Solvers : An Empirical Benchmark
- Serrin-type overdetermination in scaling limits: Sharp existence and asymptotic behavior of quasi-linear Schrödinger energy ground states
- Scaling laws for linear-elastic two-variant martensitic phase transitions in general domains
BibTeX
@misc{watanabe2026depth,
title = {{Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks}},
author = {Taishi Watanabe and Ryo Karakida and Jun-nosuke Teramae},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.35514},
eprint = {2610.35514},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.35514},
}
RIS
TY - UNPB
TI - Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks
AU - Taishi Watanabe
AU - Ryo Karakida
AU - Jun-nosuke Teramae
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2610.35514
UR - https://arxiv.org/abs/2610.35514
AB - Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the exponents quantifying how performance changes with resources. Recent theory has shown how scaling exponents are shaped by the data distribution and the structure of the learning task, but whether network depth can improve training-time scaling has remained unsettled. Through a refined analysis of deep linear networks, we clarify that the task spectrum determines when increasing network depth improves training-time scaling. For spectrally difficult tasks, increasing depth improves the asymptotic training-time scaling exponent, whereas for easy tasks, the exponent remains independent of depth, with depth affecting only a logarithmic correction at the boundary between these regimes. Remarkably, we find that a reparameterization of time governed by the predictor norm provides a unified description of the dynamics across task difficulties. Under this reparameterization, the loss follows a common scaling law whose exponent is independent of depth. Numerical experiments on a task derived from benchmark image data and with nonlinear networks exhibit qualitatively consistent task-dependent effects of depth on training-time scaling. Our findings advance the theoretical understanding of how task structure shapes the effects of network architecture on learning and offer a basis for studying scaling behavior across broader classes of neural networks.
ER -