Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons
Abstract
Hyperparameter transfer across model width can substantially reduce the cost of tuning large neural networks, but its behavior when the training horizon grows with width is not fully understood. Building on the framework of fast hyperparameter transfer (Ghosh et al., 2026), which formalizes when transfer is effective, we investigate conditions that ensure fast transfer in the growing-horizon regime. Specifically, we study learning-rate transfer in a shallow linear network with a single trainable hidden matrix, trained by full-batch gradient descent. Under additional spectral assumptions, our main results are threefold. (i) We prove fast learning-rate transfer as n,T→∞ whenever T=o(√n). (ii) We characterize the transfer rates through the finite-width perturbation scale, the first-order sensitivities of the loss and its learning-rate derivative to finite-width perturbations, and the local loss curvature. (iii) We derive limiting distributions for the optimal learning rate and optimized loss, governed by fluctuations associated with the extreme eigenvalues of the data Gram matrix. These results clarify how spectral structure and local loss sensitivities govern learning-rate transfer at growing horizons.
Related
- Learning Changes the World and Then What Was Learned has Changed: Training and Deploying Neural Networks in the Control Loop
- Algebraic Expressivity Certificates for Shallow Polynomial Neural Networks
- Fully Deterministic Shallow Neural Networks with Localized Activations. Part II: Condition Number Estimates
- Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework
- Convergence of Least Squares Methods for PDEs on R^d with Shallow ReLU^k Networks
BibTeX
@misc{sakai2026fast,
title = {{Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons}},
author = {Mana Sakai and Masaaki Imaizumi},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.35029},
eprint = {2609.35029},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.35029},
}
RIS
TY - UNPB
TI - Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons
AU - Mana Sakai
AU - Masaaki Imaizumi
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2609.35029
UR - https://arxiv.org/abs/2609.35029
AB - Hyperparameter transfer across model width can substantially reduce the cost of tuning large neural networks, but its behavior when the training horizon grows with width is not fully understood. Building on the framework of fast hyperparameter transfer (Ghosh et al., 2026), which formalizes when transfer is effective, we investigate conditions that ensure fast transfer in the growing-horizon regime. Specifically, we study learning-rate transfer in a shallow linear network with a single trainable hidden matrix, trained by full-batch gradient descent. Under additional spectral assumptions, our main results are threefold. (i) We prove fast learning-rate transfer as n,T→∞ whenever T=o(√n). (ii) We characterize the transfer rates through the finite-width perturbation scale, the first-order sensitivities of the loss and its learning-rate derivative to finite-width perturbations, and the local loss curvature. (iii) We derive limiting distributions for the optimal learning rate and optimized loss, governed by fluctuations associated with the extreme eigenvalues of the data Gram matrix. These results clarify how spectral structure and local loss sensitivities govern learning-rate transfer at growing horizons.
ER -