MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
Abstract
Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We analyze norm control in Muon over the entire training trajectory using the \emph{accumulated effective learning rate}, CT=∑t<Tηt/|Wt|F, where ηt is the nominal learning rate and ηt/|Wt|F is the effective learning rate relative to the weight norm. For bounded update directions, accumulation over any training interval bounds relative weight displacement. We show that vanilla Muon can accumulate only logarithmically, potentially limiting exploration and helping explain its slower empirical progress, whereas MuonW and MuonH guarantee at least linear accumulation under the stated conditions. Going beyond bounded-norm control, we show that sufficiently slow norm growth guarantees faster accumulation than the logarithmic worst case, opening up less restrictive optimizer designs. Based on this insight, we propose MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. With bounded update directions and a constant nominal learning rate, we prove that the weight norm grows at most proportionally to √t after t steps. Consequently, MuonGS guarantees CT=Ω(√T), with accumulation sustained even in later training. Further, for smooth scale-invariant objectives we establish its O(T−1/2) stationarity guarantee. In pretraining experiments with 160M- and 400M-parameter dense models as well as a 135M active- and 1B total- parameter MoE model, MuonGS achieves lower validation perplexity than MuonH at long training horizons. In particular, for the dense model it achieves an estimated token-equivalent speedup up to about 15% at matched validation loss to MuonH.
Related
- Finiteness of singularities for higher-weight axially symmetric minimizers of the relaxed Dirichlet energy
- Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems
- Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting
- Harnessing LLMs with Direction-Scope Control for Combinatorial Solver Refinement
- Risk-matched case-control sampling and semiparametric weight calibration for efficient biomarker assessment with binary outcomes
BibTeX
@misc{glentis2026muongs,
title = {{MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control}},
author = {Athanasios Glentis and Dawei Li and Mingyi Hong},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.37647},
eprint = {2610.37647},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.37647},
}
RIS
TY - UNPB
TI - MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
AU - Athanasios Glentis
AU - Dawei Li
AU - Mingyi Hong
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2610.37647
UR - https://arxiv.org/abs/2610.37647
AB - Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We analyze norm control in Muon over the entire training trajectory using the \emph{accumulated effective learning rate}, C_T=∑_t<Tη_t/|W_t|_F, where η_t is the nominal learning rate and η_t/|W_t|_F is the effective learning rate relative to the weight norm. For bounded update directions, accumulation over any training interval bounds relative weight displacement. We show that vanilla Muon can accumulate only logarithmically, potentially limiting exploration and helping explain its slower empirical progress, whereas MuonW and MuonH guarantee at least linear accumulation under the stated conditions. Going beyond bounded-norm control, we show that sufficiently slow norm growth guarantees faster accumulation than the logarithmic worst case, opening up less restrictive optimizer designs. Based on this insight, we propose MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. With bounded update directions and a constant nominal learning rate, we prove that the weight norm grows at most proportionally to √t after t steps. Consequently, MuonGS guarantees C_T=Ω(√T), with accumulation sustained even in later training. Further, for smooth scale-invariant objectives we establish its O(T^−1/2) stationarity guarantee. In pretraining experiments with 160M- and 400M-parameter dense models as well as a 135M active- and 1B total- parameter MoE model, MuonGS achieves lower validation perplexity than MuonH at long training horizons. In particular, for the dense model it achieves an estimated token-equivalent speedup up to about 15% at matched validation loss to MuonH.
ER -