Sharpness-aware Orthogonalized Momentum Algorithms

Abstract

Adversarial sharpness penalty per iteration of a base optimizer, known as sharpness-aware minimization (SAM), is an accepted technique to improve generalization in deep learning. Recently, Muon has emerged as a dominant optimization algorithm for training large language models (LLMs). It achieves lower sample complexity for heavy-tailed gradient updates - a phenomenon common in LLM training. However, sharpness penalties with Muon updates are yet largely unexplored in the literature. Moreover, the behavior of SAM under heavy-tailed noise is unknown both theoretically and empirically. A potential reason for this gap is the additional cost of extra gradient computation per iteration for penalty perturbations in SAM, which becomes prohibitive for LLMs. This paper introduces Sharpness-aware Orthogonalized Momentum Algorithm (SOMA) to combine a computationally efficient and theoretically grounded perturbation scheme with the Muon algorithm. The key idea is to use stale (purely) stochastic gradient noise for adversarial perturbations. Theoretically, we show that SOMA's sample complexity is invariant to the perturbation direction. We provide explorations and insights into SOMA's robustness to heavy-tailed gradient noise. We extensively explore SOMA's efficacy in (a) mitigating quality degradation due to post-training quantization, and (b) catastrophic forgetting during fine-tuning of LLMs.

Infeasible Momentum Orthogonalized Methods for Optimization on the Stiefel Manifold
Yuxuan Zeng, Qi Deng, Dongdong Ge +1
arXiv30 Oct 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Nonsmooth Optimization via Orthogonalized Momentum
Lexiao Lai, Tianyi Lin, Jiayu Zhang
arXiv9 Oct 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Sharpness-Aware Minimization (SAM) Improves Classification Accuracy of Bacterial Raman Spectral Data Enabling Portable Diagnostics
Kaitlin Zareno, Jarett Dewbury, Siamak K. Sorooshyari +2
arXiv16 Sep 2026PreprintCC BY 4.0Full textBioinformatics & computational biology StatisticsarXiv
Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization
Nour Jamoussi, Marios Kountouris
arXiv8 Sep 2026PreprintCC BY 4.0Full textGeneral mathematicsarXiv
Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness
Dimitris Oikonomou, Nicolas Loizou
arXiv21 Oct 2026PreprintCC BY 4.0Full textOptimisation & control StatisticsarXiv

Cite

BibTeX RIS

BibTeX

@misc{geetika2026sharpnessaware,
  title = {{Sharpness-aware Orthogonalized Momentum Algorithms}},
  author = {Geetika and Shivani Singh Chauhan and Gauranshi Gupta and Anant Jain and Kaushik Kalyanaraman and Bapi Chatterjee},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.32465},
  eprint = {2610.32465},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32465},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Sharpness-aware Orthogonalized Momentum Algorithms
AU  - Geetika
AU  - Shivani Singh Chauhan
AU  - Gauranshi Gupta
AU  - Anant Jain
AU  - Kaushik Kalyanaraman
AU  - Bapi Chatterjee
PY  - 2026
DA  - 2026/10/23
PB  - arXiv
DO  - 10.48550/arxiv.2610.32465
UR  - https://arxiv.org/abs/2610.32465
AB  - Adversarial sharpness penalty per iteration of a base optimizer, known as sharpness-aware minimization (SAM), is an accepted technique to improve generalization in deep learning. Recently, Muon has emerged as a dominant optimization algorithm for training large language models (LLMs). It achieves lower sample complexity for heavy-tailed gradient updates - a phenomenon common in LLM training. However, sharpness penalties with Muon updates are yet largely unexplored in the literature. Moreover, the behavior of SAM under heavy-tailed noise is unknown both theoretically and empirically. A potential reason for this gap is the additional cost of extra gradient computation per iteration for penalty perturbations in SAM, which becomes prohibitive for LLMs. This paper introduces Sharpness-aware Orthogonalized Momentum Algorithm (SOMA) to combine a computationally efficient and theoretically grounded perturbation scheme with the Muon algorithm. The key idea is to use stale (purely) stochastic gradient noise for adversarial perturbations. Theoretically, we show that SOMA's sample complexity is invariant to the perturbation direction. We provide explorations and insights into SOMA's robustness to heavy-tailed gradient noise. We extensively explore SOMA's efficacy in (a) mitigating quality degradation due to post-training quantization, and (b) catastrophic forgetting during fine-tuning of LLMs.
ER  -

Abstract

Related

Cite