On the Comparison of Optimizers for Imbalanced Learning
Abstract
Data imbalance is pervasive in machine learning, from rare words and anomalies to underrepresented patterns in heterogeneous or cross-tabulated data. We study idealized optimizers geometries in continuous time to model small-step training in deep learning. We assume that the source of imbalance is unobserved: the optimizer has only access to the aggregate training loss ignoring the exact contributions of the majority and minority groups. In this setting, we characterize a region where majority losses are optimized regardless of the admissible minority structure. We derive explicit equations of this zone and bounds on the time needed to leave it. These bounds exhibit a milder dependence on minority amplitude for sign, spectral, and Newton descent than for Euclidean gradient descent. Experiments with AdamW and Muon suggest similar advantages over SGD across language, tabular, and image tasks.
Related
- Large-scale empirical tuning and comparison of default optimizers for variational inference
- Hierarchical Majority Decomposition for Imbalanced Classification: A Recursive Binary Majority-Class Tree with Persistent Minority Comparison
- Reinforcement learning to choose optimizers
- ML-CLAPPS: A Probabilistic Framework for Imbalanced Multilabel Learning under Class Prioritization
- A Fuzzy Adjustment Mechanism to Tune Focal Loss Hyperparameters in Imbalanced Deep Learning
BibTeX
@misc{lezane2026comparison,
title = {{On the Comparison of Optimizers for Imbalanced Learning}},
author = {Clément Lezane and François Bachoc and Jérôme Bolte and Jean-Michel Loubes},
howpublished = {HAL},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06421},
eprint = {2610.06421},
archivePrefix = {arXiv},
url = {https://hal.science/hal-05775266v1},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - On the Comparison of Optimizers for Imbalanced Learning
AU - Clément Lezane
AU - François Bachoc
AU - Jérôme Bolte
AU - Jean-Michel Loubes
PY - 2026
DA - 2026/10/02
PB - HAL
DO - 10.48550/arxiv.2610.06421
UR - https://hal.science/hal-05775266v1
AB - Data imbalance is pervasive in machine learning, from rare words and anomalies to underrepresented patterns in heterogeneous or cross-tabulated data. We study idealized optimizers geometries in continuous time to model small-step training in deep learning. We assume that the source of imbalance is unobserved: the optimizer has only access to the aggregate training loss ignoring the exact contributions of the majority and minority groups. In this setting, we characterize a region where majority losses are optimized regardless of the admissible minority structure. We derive explicit equations of this zone and bounds on the time needed to leave it. These bounds exhibit a milder dependence on minority amplitude for sign, spectral, and Newton descent than for Euclidean gradient descent. Experiments with AdamW and Muon suggest similar advantages over SGD across language, tabular, and image tasks.
ER -