Regularity-Aware Stochastic MGDA with Adaptive Per-Iterate Conflict-Avoidant Update
Abstract
Multi-objective learning (MOL) aims to optimize multiple objectives simultaneously. The multi-gradient descent algorithm (MGDA) constructs a common descent or conflict-avoidant (CA) direction by solving a gradient-aggregation subproblem. In stochastic settings, the vanilla stochastic MGDA method, SMG, replaces full-batch gradients with mini-batch estimates, causing bias in the resulting CA direction. Existing analyses control this error using continuity of the CA-direction mapping, but do not distinguish how its sensitivity varies with the geometry of the MGDA subproblem. Motivated by this limitation, we show that the MGDA direction map is Lipschitz continuous under reduced nondegeneracy, whereas its general 1/2-Hölder exponent on bounded sets is sharp. This geometry motivates the Multi-objective Regularity-aware (MoRe) method, which retains adaptive MGDA aggregation in the regular regime and switches to a fixed-weight linear scalarization (LS) near degeneracy. The fallback branch skips the MGDA subproblem solve, which can reduce aggregation cost on fallback iterations while avoiding a potentially noise-sensitive update. Theoretically, we establish a per-iterate CA guarantee, and a 𝒪̃(T−1) convergence rate, improving upon the previously established 𝒪̃(T−1/4) SMG guarantee, where 𝒪̃ hides logarithmic factors. Experiments demonstrate favorable multi-task performance and convergence behavior consistent with the theoretical rate.
Related
- The Last Iterate of the Projected Stochastic Subgradient Method Pays a Logarithm, Even in Dimension One
- Last-Iterate Convergence of Stochastic Gradient Methods for Unbounded Problems
- Improving the Last-Iterate Guarantees of Anytime Algorithms for Stochastic Monotone Variational Inequalities
- Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
- Stability of negative-regularity mixing and applications to stochastic fluids
BibTeX
@misc{huang2026regularityaware,
title = {{Regularity-Aware Stochastic MGDA with Adaptive Per-Iterate Conflict-Avoidant Update}},
author = {Chentong Huang and Lisha Chen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2607.15412},
eprint = {2607.15412},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.15412},
}
RIS
TY - UNPB
TI - Regularity-Aware Stochastic MGDA with Adaptive Per-Iterate Conflict-Avoidant Update
AU - Chentong Huang
AU - Lisha Chen
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2607.15412
UR - https://arxiv.org/abs/2607.15412
AB - Multi-objective learning (MOL) aims to optimize multiple objectives simultaneously. The multi-gradient descent algorithm (MGDA) constructs a common descent or conflict-avoidant (CA) direction by solving a gradient-aggregation subproblem. In stochastic settings, the vanilla stochastic MGDA method, SMG, replaces full-batch gradients with mini-batch estimates, causing bias in the resulting CA direction. Existing analyses control this error using continuity of the CA-direction mapping, but do not distinguish how its sensitivity varies with the geometry of the MGDA subproblem. Motivated by this limitation, we show that the MGDA direction map is Lipschitz continuous under reduced nondegeneracy, whereas its general 1/2-Hölder exponent on bounded sets is sharp. This geometry motivates the Multi-objective Regularity-aware (MoRe) method, which retains adaptive MGDA aggregation in the regular regime and switches to a fixed-weight linear scalarization (LS) near degeneracy. The fallback branch skips the MGDA subproblem solve, which can reduce aggregation cost on fallback iterations while avoiding a potentially noise-sensitive update. Theoretically, we establish a per-iterate CA guarantee, and a 𝒪̃(T^−1) convergence rate, improving upon the previously established 𝒪̃(T^−1/4) SMG guarantee, where 𝒪̃ hides logarithmic factors. Experiments demonstrate favorable multi-task performance and convergence behavior consistent with the theoretical rate.
ER -