Anytime Training with Schedule-Free Spectral Optimization
Abstract
Standard neural network training relies on learning-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re-tuning as data availability changes. Schedule-Free (SF) methods address this by removing explicit schedules, yet SF-AdamW, the current state-of-the-art anytime optimizer, consistently underperforms well-tuned AdamW baselines. We propose SF-NorMuon, a schedule-free spectral optimizer that closes this gap: with a single hyperparameter configuration, SF-NorMuon matches or exceeds tuned AdamW on 125M and 772M parameter language models across $1−−8\times$ Chinchilla horizons, and trails horizon-aware, cosine-scheduled NorMuon by only ∼ 0.03 nats, substantially narrowing the gap between anytime and scheduled spectral optimizers. On the theoretical side, we prove a stationarity guarantee for schedule-free spectral dynamics and identify weight decay at the fast iterate as essential for long-horizon stability. SF-NorMuon enables practitioners to obtain high-quality checkpoints at any point during training without committing to a horizon in advance. Fully matching horizon-aware spectral optimizers without a schedule remains an open problem, which we highlight as a direction for future work. By closing the performance gap with tuned AdamW baselines, SF-NorMuon makes horizon-free optimization more practical, taking a step towards truly open-ended, continual learning.
Related
- On the Performance Evaluation of Anytime Algorithms in Noisy Optimization
- Schedule optimization for tau-leaping in masked discrete diffusion
- Time-reliability optimization of multi-pass underfill dispensing sequences on printed circuit boards: a schedule-dependent reliability model solved by a multiset-permutation binary-addition-tree algorithm
- Collision-Free Schedule Acquisition through Slotted ALOHA: Scaling Laws and Fluctuations
- Anytime-Feasible Gradient Descent for Constrained Optimization Under Gradient Uncertainty
BibTeX
@misc{apte2026anytime,
title = {{Anytime Training with Schedule-Free Spectral Optimization}},
author = {Anuj Apte and Pranav Deshpande and Niraj Kumar and Shouvanik Chakrabarti and Junhyung Lyle Kim},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.23061},
eprint = {2605.23061},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.23061},
}
RIS
TY - UNPB
TI - Anytime Training with Schedule-Free Spectral Optimization
AU - Anuj Apte
AU - Pranav Deshpande
AU - Niraj Kumar
AU - Shouvanik Chakrabarti
AU - Junhyung Lyle Kim
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2605.23061
UR - https://arxiv.org/abs/2605.23061
AB - Standard neural network training relies on learning-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re-tuning as data availability changes. Schedule-Free (SF) methods address this by removing explicit schedules, yet SF-AdamW, the current state-of-the-art anytime optimizer, consistently underperforms well-tuned AdamW baselines. We propose SF-NorMuon, a schedule-free spectral optimizer that closes this gap: with a single hyperparameter configuration, SF-NorMuon matches or exceeds tuned AdamW on 125M and 772M parameter language models across $1−−8\times$ Chinchilla horizons, and trails horizon-aware, cosine-scheduled NorMuon by only ∼ 0.03 nats, substantially narrowing the gap between anytime and scheduled spectral optimizers. On the theoretical side, we prove a stationarity guarantee for schedule-free spectral dynamics and identify weight decay at the fast iterate as essential for long-horizon stability. SF-NorMuon enables practitioners to obtain high-quality checkpoints at any point during training without committing to a horizon in advance. Fully matching horizon-aware spectral optimizers without a schedule remains an open problem, which we highlight as a direction for future work. By closing the performance gap with tuned AdamW baselines, SF-NorMuon makes horizon-free optimization more practical, taking a step towards truly open-ended, continual learning.
ER -