Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness

Abstract

Sharpness-Aware Minimization (SAM) has established itself as a widely used method for training deep neural networks. Despite its popularity, existing convergence guarantees for SAM-type methods require smoothness, a restrictive assumption. Generalized smoothness conditions have recently been proposed in the analysis of gradient methods but remain underexplored for SAM. To analyze the effect of SAM's perturbation under generalized smoothness, we use relative gradient distortion to quantify how the sharpness-aware direction departs from the original gradient. Under (L₀,L₁)-smoothness, we show that this distortion can become unbounded near stationarity for SAM and at large gradients for its unnormalized variant (USAM), with examples where SAM cycles and USAM diverges. We then establish a necessary condition for uniform distortion control among perturbations along the gradient whose nonnegative multipliers depend only on its norm. Motivated by this condition, we propose Curvature-Scaled SAM (CS-SAM), which scales both the perturbation and the descent step by suitable curvature upper bounds, and prove that it uniformly bounds the distortion. Under both (L₀,L₁)- and (H₀,H₁)-smoothness, we establish deterministic 𝒪(1/T) rates for the best squared gradient norm on nonconvex objectives, last-iterate 𝒪(1/T) rates for the function gap on convex objectives with a minimizer, and linear convergence under the Polyak-Lojasiewicz condition. With zero perturbation, our guarantees recover the rates for gradient descent with (L₀,L₁)- or (H₀,H₁)-based step-sizes. Minibatch experiments on computer vision tasks demonstrate competitive performance for both variants, with the (H₀,H₁) variant achieving the highest mean best-test accuracy in all four settings against tuned SAM and USAM with constant and cosine annealing schedules.

Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization
Nour Jamoussi, Marios Kountouris
arXiv8 Sep 2026PreprintCC BY 4.0Full textGeneral mathematicsarXiv
Sharpness-Aware Minimization (SAM) Improves Classification Accuracy of Bacterial Raman Spectral Data Enabling Portable Diagnostics
Kaitlin Zareno, Jarett Dewbury, Siamak K. Sorooshyari +2
arXiv16 Sep 2026PreprintCC BY 4.0Full textBioinformatics & computational biology StatisticsarXiv
Global Linear Convergence of the Proximal Bundle Method under Unknown Piecewise Smoothness and Quadratic Growth
Zhenwei Lin, Zhe Zhang
arXiv10 Sep 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness
Yuki Takezawa, Eduard Gorbunov
arXiv5 Oct 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and Hölder Smoothness
Misbah Uz Zaman, Anirbit Mukherjee
arXiv11 Sep 2026PreprintCC BY-NC-SA 4.0Full textOptimisation & control StatisticsarXiv

Cite

BibTeX RIS

BibTeX

@misc{oikonomou2026curvaturescaled,
  title = {{Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness}},
  author = {Dimitris Oikonomou and Nicolas Loizou},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.29192},
  eprint = {2610.29192},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.29192},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness
AU  - Dimitris Oikonomou
AU  - Nicolas Loizou
PY  - 2026
DA  - 2026/10/21
PB  - arXiv
DO  - 10.48550/arxiv.2610.29192
UR  - https://arxiv.org/abs/2610.29192
AB  - Sharpness-Aware Minimization (SAM) has established itself as a widely used method for training deep neural networks. Despite its popularity, existing convergence guarantees for SAM-type methods require smoothness, a restrictive assumption. Generalized smoothness conditions have recently been proposed in the analysis of gradient methods but remain underexplored for SAM. To analyze the effect of SAM's perturbation under generalized smoothness, we use relative gradient distortion to quantify how the sharpness-aware direction departs from the original gradient. Under (L_0,L_1)-smoothness, we show that this distortion can become unbounded near stationarity for SAM and at large gradients for its unnormalized variant (USAM), with examples where SAM cycles and USAM diverges. We then establish a necessary condition for uniform distortion control among perturbations along the gradient whose nonnegative multipliers depend only on its norm. Motivated by this condition, we propose Curvature-Scaled SAM (CS-SAM), which scales both the perturbation and the descent step by suitable curvature upper bounds, and prove that it uniformly bounds the distortion. Under both (L_0,L_1)- and (H_0,H_1)-smoothness, we establish deterministic 𝒪(1/T) rates for the best squared gradient norm on nonconvex objectives, last-iterate 𝒪(1/T) rates for the function gap on convex objectives with a minimizer, and linear convergence under the Polyak-Lojasiewicz condition. With zero perturbation, our guarantees recover the rates for gradient descent with (L_0,L_1)- or (H_0,H_1)-based step-sizes. Minibatch experiments on computer vision tasks demonstrate competitive performance for both variants, with the (H_0,H_1) variant achieving the highest mean best-test accuracy in all four settings against tuned SAM and USAM with constant and cosine annealing schedules.
ER  -

Abstract

Related

Cite