Does Muon Need Fine-Grained Spectral Shaping?
Abstract
Muon combines current and past gradients into matrix momentum. For M=UΣV⊤, the idealized polar update Q=UV⊤ gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately $94−−97\%$ of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko--Pastur reference edge for Muon's Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk's relative weight through one shared gain while preserving the Frobenius norm of each matrix's unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. Across 30 continued-pretraining settings spanning Pythia-14M to 410M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon and outperforms Spectra. Measured against Muon's flat profile, Freon reduces final loss by 0.022% of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of $0.073−−0.147\%$. These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles.
Related
- Broadcasting in Centimeter-Wave Gigantic MIMO Systems: Beam Shaping is All You Need
- How Does Probabilistic Constellation Shaping Control Ranging Sidelobe Levels in ISAC?
- How Many Labels Does Model Choice Need? Certificates and Budgets for Selective Prediction
- When does conformal calibration need censoring weights? Cause-of-failure prediction sets under competing risks
- Spectral Deflation for Factorization-Free Matrix Filtering in Muon and Semidefinite Programming
BibTeX
@misc{chaitanya2026does,
title = {{Does Muon Need Fine-Grained Spectral Shaping?}},
author = {Meher Chaitanya and Tianyi Zhou and Aristides Gionis},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.07497},
eprint = {2610.07497},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07497},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Does Muon Need Fine-Grained Spectral Shaping?
AU - Meher Chaitanya
AU - Tianyi Zhou
AU - Aristides Gionis
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.07497
UR - https://arxiv.org/abs/2610.07497
AB - Muon combines current and past gradients into matrix momentum. For M=UΣV^⊤, the idealized polar update Q=UV^⊤ gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately $94−−97\%$ of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko--Pastur reference edge for Muon's Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk's relative weight through one shared gain while preserving the Frobenius norm of each matrix's unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. Across 30 continued-pretraining settings spanning Pythia-14M to 410M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon and outperforms Spectra. Measured against Muon's flat profile, Freon reduces final loss by 0.022% of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of $0.073−−0.147\%$. These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles.
ER -