Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization

Abstract

Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning, both motivated by robustness to perturbations. However, their relationship has remained largely unexplored. Building on classical second-order expansions of f-divergences, we show that the two methods are locally consistent under parameter-space perturbations: both induce curvature-sensitive penalties, with divergence regularization yielding a Fisher-weighted quadratic form and SAM penalizing sharpness through the dominant Hessian eigenvalue. For negative log-likelihood objectives with exponential-family output distributions, this correspondence becomes especially transparent, since the Fisher and Gauss-Newton matrices coincide. We further show that the same local geometric perspective extends to input-space perturbations, where divergence-based regularization is defined through transformations of the input. In this setting, the regularizer induces a pullback quadratic form on the input space, providing a more general perturbation framework than standard SAM while preserving the same local sensitivity interpretation. To validate the analysis empirically, we use the asymmetric α-skew Jensen-Shannon divergence (JSD) family as a controlled testbed. Its local curvature coefficient scales as α(1−α) and is maximized at the symmetric point α=1/2, which recovers the standard JSD. Loss-landscape visualizations in the input-perturbation regime show that stronger induced curvature penalization is associated with flatter local minima. Experiments on four benchmark datasets further demonstrate that both accuracy and negative log-likelihood are consistently best near this regime of maximal curvature penalization.

Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness
Dimitris Oikonomou, Nicolas Loizou
arXiv21 Oct 2026PreprintCC BY 4.0Full textOptimisation & control StatisticsarXiv
Sharpness-Aware Minimization (SAM) Improves Classification Accuracy of Bacterial Raman Spectral Data Enabling Portable Diagnostics
Kaitlin Zareno, Jarett Dewbury, Siamak K. Sorooshyari +2
arXiv16 Sep 2026PreprintCC BY 4.0Full textBioinformatics & computational biology StatisticsarXiv
Equivalence Between Continuous-Time Risk-Sensitive Control and Rényi Divergence Minimization
Shinji Kataoka, Kaoru Teranishi, Yasumasa Fujisaki
arXiv7 Sep 2026PreprintCC BY 4.0Full textOptimisation & controlarXiv
Geodesics with Unified Tangent-constrained Priors and Curvature Regularization
Chong Di, Li Liu, Jinglin Zhang +3
HAL28 May 2026PreprintLicence not statedNumerical analysis & scientific computingHAL
Fractional-order differential geometry of Gielis curves: chart regularization, Caputo curvature, and the fate of singularities
Umut Selvi, Derya Sağlam
arXiv1 Oct 2026PreprintCC BY 4.0Full textGeneral mathematicsarXiv

Cite

BibTeX RIS

BibTeX

@misc{jamoussi2026explaining,
  title = {{Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization}},
  author = {Nour Jamoussi and Marios Kountouris},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.09367},
  eprint = {2609.09367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.09367},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization
AU  - Nour Jamoussi
AU  - Marios Kountouris
PY  - 2026
DA  - 2026/09/08
PB  - arXiv
DO  - 10.48550/arxiv.2609.09367
UR  - https://arxiv.org/abs/2609.09367
AB  - Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning, both motivated by robustness to perturbations. However, their relationship has remained largely unexplored. Building on classical second-order expansions of f-divergences, we show that the two methods are locally consistent under parameter-space perturbations: both induce curvature-sensitive penalties, with divergence regularization yielding a Fisher-weighted quadratic form and SAM penalizing sharpness through the dominant Hessian eigenvalue. For negative log-likelihood objectives with exponential-family output distributions, this correspondence becomes especially transparent, since the Fisher and Gauss-Newton matrices coincide. We further show that the same local geometric perspective extends to input-space perturbations, where divergence-based regularization is defined through transformations of the input. In this setting, the regularizer induces a pullback quadratic form on the input space, providing a more general perturbation framework than standard SAM while preserving the same local sensitivity interpretation. To validate the analysis empirically, we use the asymmetric α-skew Jensen-Shannon divergence (JSD) family as a controlled testbed. Its local curvature coefficient scales as α(1−α) and is maximized at the symmetric point α=1/2, which recovers the standard JSD. Loss-landscape visualizations in the input-perturbation regime show that stronger induced curvature penalization is associated with flatter local minima. Experiments on four benchmark datasets further demonstrate that both accuracy and negative log-likelihood are consistently best near this regime of maximal curvature penalization.
ER  -

Abstract

Related

Cite