From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon
Abstract
Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether this difference provably affects population performance remains unclear. We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam, a proxy for random-reshuffling Adam, and exact-SVD Muon in high-dimensional multiclass classification. Under an isotropic Gaussian-cloud data model, this advantage arises because row normalization's class-wise Euclidean geometry asymptotically preserves the population decision-boundary directions, whereas Adam's coordinate-wise geometry and Muon's spectral geometry introduce nonvanishing distortions. Beyond isotropy, the advantage persists for full-batch training on class means with independently oriented class-mean and test-noise covariances. It holds for power-law spectra with class-mean exponent below one, even under heavily anisotropic test noise. When both covariances are diagonal and sufficiently close, the advantage over Adam can reverse, while applying the same random rotation to both restores it by changing only their alignment with Adam's coordinate axes. Synthetic and last-layer language-model experiments support the predicted advantage.
Related
- A Statistical Viewpoint on Adam, Shampoo and Muon
- The Row Normalization Puzzle in Muon
- Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias
- Embedding Geometry Shapes LLM Routing: Tracing Failure Modes and Improving Generalization
- When Is Muon Optimal? Radial Fragility and Minimax Geometry
BibTeX
@misc{kim2026from,
title = {{From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon}},
author = {Jihwan Kim and Dogyoon Song and Chulhee Yun},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.11309},
eprint = {2610.11309},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.11309},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon
AU - Jihwan Kim
AU - Dogyoon Song
AU - Chulhee Yun
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.11309
UR - https://arxiv.org/abs/2610.11309
AB - Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether this difference provably affects population performance remains unclear. We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam, a proxy for random-reshuffling Adam, and exact-SVD Muon in high-dimensional multiclass classification. Under an isotropic Gaussian-cloud data model, this advantage arises because row normalization's class-wise Euclidean geometry asymptotically preserves the population decision-boundary directions, whereas Adam's coordinate-wise geometry and Muon's spectral geometry introduce nonvanishing distortions. Beyond isotropy, the advantage persists for full-batch training on class means with independently oriented class-mean and test-noise covariances. It holds for power-law spectra with class-mean exponent below one, even under heavily anisotropic test noise. When both covariances are diagonal and sufficiently close, the advantage over Adam can reverse, while applying the same random rotation to both restores it by changing only their alignment with Adam's coordinate axes. Synthetic and last-layer language-model experiments support the predicted advantage.
ER -