Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization
Abstract
The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this transformation in feature learning remain poorly understood. In this work, we investigate this question through a tractable factual-recall model, where a fact maps each subject-relation pair to an answer, and a linear transformer learns the subject- and relation-dependent information required to recover this mapping. The transformer is optimized with gradient flow (GF), spectral GF, or Sign GF, which are continuous-time limits of gradient descent, Muon, and Adam, respectively. Prior studies (Nichani et al., 2025) have shown that when the number of subjects exceeds the number of relations, GF learns relation-dependent information before subject-dependent information, producing a feature-separation phase during training. We characterize this separation with the learning times when the subject- and relation-dependent components of the prediction reach a target accuracy. With S subjects and R relations, GF has a learning-time ratio of Θ̃(√(S/R)), whereas Spectral GF reduces this ratio to Θ̃(1). In addition, for fixed S and R, the subject- and relation-dependent errors decay as 1/(Tlog T) in training time T under GF, but as exp(−poly(T)) under spectral GF. Finally, we show that GF and spectral GF are equivariant under orthogonal transformations of the token embeddings, whereas Sign GF is not: Different orthonormal embeddings can potentially produce no feature separation, a large feature-separation phase, or even a reversed learning order. These results provide a mechanistic view of how spectral orthogonalization can fundamentally reshape feature-learning dynamics.
Related
- Understanding Latent-Dimension Scaling in Dynamical-System Learning through Spectral Reliability
- Simultaneous Clustered Orthogonalization
- Physical Muon: Orthogonalization as an Equilibrium Computation
- Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization
- Polar Switch: Fast Muon Orthogonalization with Polynomial Circuits
BibTeX
@misc{li2026muon,
title = {{Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization}},
author = {Xuheng Li and Qiwei Di and Yuan Cao and Quanquan Gu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02798},
eprint = {2610.02798},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02798},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization
AU - Xuheng Li
AU - Qiwei Di
AU - Yuan Cao
AU - Quanquan Gu
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2610.02798
UR - https://arxiv.org/abs/2610.02798
AB - The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this transformation in feature learning remain poorly understood. In this work, we investigate this question through a tractable factual-recall model, where a fact maps each subject-relation pair to an answer, and a linear transformer learns the subject- and relation-dependent information required to recover this mapping. The transformer is optimized with gradient flow (GF), spectral GF, or Sign GF, which are continuous-time limits of gradient descent, Muon, and Adam, respectively. Prior studies (Nichani et al., 2025) have shown that when the number of subjects exceeds the number of relations, GF learns relation-dependent information before subject-dependent information, producing a feature-separation phase during training. We characterize this separation with the learning times when the subject- and relation-dependent components of the prediction reach a target accuracy. With S subjects and R relations, GF has a learning-time ratio of Θ̃(√(S/R)), whereas Spectral GF reduces this ratio to Θ̃(1). In addition, for fixed S and R, the subject- and relation-dependent errors decay as 1/(Tlog T) in training time T under GF, but as exp(−poly(T)) under spectral GF. Finally, we show that GF and spectral GF are equivariant under orthogonal transformations of the token embeddings, whereas Sign GF is not: Different orthonormal embeddings can potentially produce no feature separation, a large feature-separation phase, or even a reversed learning order. These results provide a mechanistic view of how spectral orthogonalization can fundamentally reshape feature-learning dynamics.
ER -