Convergence of Practical Muon
Abstract
Muon is emerging as a promising alternative to AdamW for large-scale neural network training, yet theoretical understanding of its practical implementation remains incomplete, as existing analyses often simplify or omit two key components: (i) practical Newton--Schulz iterations with empirically tuned polynomial coefficients $(3.4445,-4.7750,2.0315)$; and (ii) decoupled weight decay for regularization. In this paper, we provide an optimization interpretation and establish convergence for practical Muon, jointly accounting for both components. Specifically, we interpret practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted ℓ2 regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective. We then establish, to our best knowledge, the first convergence guarantee for practical Muon in the stochastic nonconvex setting, with an 𝒪(T−1/4) convergence rate in terms of the expected Frobenius norm of the gradient, improving the dimension dependence of the best known AdamW's convergence rate by a factor of √d, where T is the iteration horizon and d is the parameter dimension. Experiments further support the theoretical convergence results.
Related
- Convergence guarantees for Muon: New parameter regimes and generalizations
- MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
- Muon-C: Operator-Aligned Muon for Convolutional Kernels
- Spacetimes of maximal proper acceleration and muon mean lifetime
- Development of a Physics-Informed Neural Framework, MEOWN, for Rapid Prediction of Muon Stopping Sites in Crystalline Materials, for understanding Quantum Magnet employing Muon Spectroscopy
BibTeX
@misc{wang2026convergence,
title = {{Convergence of Practical Muon}},
author = {Haonan Wang and Yu Wu and Minghui Liwang and Xinlei Yi and Yiguang Hong},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.33152},
eprint = {2609.33152},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.33152},
}
RIS
TY - UNPB
TI - Convergence of Practical Muon
AU - Haonan Wang
AU - Yu Wu
AU - Minghui Liwang
AU - Xinlei Yi
AU - Yiguang Hong
PY - 2026
DA - 2026/09/27
PB - arXiv
DO - 10.48550/arxiv.2609.33152
UR - https://arxiv.org/abs/2609.33152
AB - Muon is emerging as a promising alternative to AdamW for large-scale neural network training, yet theoretical understanding of its practical implementation remains incomplete, as existing analyses often simplify or omit two key components: (i) practical Newton--Schulz iterations with empirically tuned polynomial coefficients $(3.4445,-4.7750,2.0315)$; and (ii) decoupled weight decay for regularization. In this paper, we provide an optimization interpretation and establish convergence for practical Muon, jointly accounting for both components. Specifically, we interpret practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted ℓ_2 regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective. We then establish, to our best knowledge, the first convergence guarantee for practical Muon in the stochastic nonconvex setting, with an 𝒪(T^−1/4) convergence rate in terms of the expected Frobenius norm of the gradient, improving the dimension dependence of the best known AdamW's convergence rate by a factor of √d, where T is the iteration horizon and d is the parameter dimension. Experiments further support the theoretical convergence results.
ER -