SIMS: Scale-Invariant Merit-Function-Based Scalarization for Multi-Task Learning
Abstract
Multi-task learning (MTL) requires navigating unavoidable trade-offs among competing objectives. This paradigm is frequently formulated as multi-objective optimization (MOO), where the scalarization is favored to reduce an MOO problem to a single objective. We empirically find that existing merit-function-based scalarization approaches are sensitive to the relative scales of different objectives in practical MTL, where task losses commonly differ by orders of magnitude. The optimization process often favors objectives with larger scales even though the underlying Pareto optimal solutions remains invariant to rescaling (i.e., multiplying an objective by a positive constant). To address this issue, we propose Scale-Invariant Merit-function-based Scalarization (SIMS) for MTL. Specifically, SIMS adopts a transformation-induced merit function to convert the MOO problem of MTL to a single objective that renders optimization invariant to the magnitudes of losses. Theoretically, we prove that the requirement for scale invariance uniquely determines this transformation to be logarithmic. We further show that this general transformation-induced merit function preserves weak Pareto optimality and admits a smooth surrogate with controllable approximation error. Extensive experiments on representative multi-task benchmarks demonstrate that SIMS consistently outperforms existing scalarization methods and achieves state-of-the-art performance.
Versions
Related
- Explicit height bounds on modular polynomials for the elliptic j-invariant, cube root of j, and Weber modular function f
- Scale-invariant bounds on thin sets for measures satisfying a first-order PDE and the anisotropic Michael-Simon inequality
- Unbalanced Minimal Matchings for Measurable Scale Invariant Costs on ℝd
- Regularity of critical immersions for scale-invariant curvature energies in even dimensions
- Unbiased estimation of normalized scale-invariant indices under the Gamma distribution
BibTeX
@misc{chen2026sims,
title = {{SIMS: Scale-Invariant Merit-Function-Based Scalarization for Multi-Task Learning}},
author = {Zebin Chen and Fei Xing and Yang Chen and Hua Liu and Andy HF Chow and Yuhua Qian and Yu Zhang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.12599},
eprint = {2609.12599},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.12599},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - SIMS: Scale-Invariant Merit-Function-Based Scalarization for Multi-Task Learning
AU - Zebin Chen
AU - Fei Xing
AU - Yang Chen
AU - Hua Liu
AU - Andy HF Chow
AU - Yuhua Qian
AU - Yu Zhang
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2609.12599
UR - https://arxiv.org/abs/2609.12599
AB - Multi-task learning (MTL) requires navigating unavoidable trade-offs among competing objectives. This paradigm is frequently formulated as multi-objective optimization (MOO), where the scalarization is favored to reduce an MOO problem to a single objective. We empirically find that existing merit-function-based scalarization approaches are sensitive to the relative scales of different objectives in practical MTL, where task losses commonly differ by orders of magnitude. The optimization process often favors objectives with larger scales even though the underlying Pareto optimal solutions remains invariant to rescaling (i.e., multiplying an objective by a positive constant). To address this issue, we propose Scale-Invariant Merit-function-based Scalarization (SIMS) for MTL. Specifically, SIMS adopts a transformation-induced merit function to convert the MOO problem of MTL to a single objective that renders optimization invariant to the magnitudes of losses. Theoretically, we prove that the requirement for scale invariance uniquely determines this transformation to be logarithmic. We further show that this general transformation-induced merit function preserves weak Pareto optimality and admits a smooth surrogate with controllable approximation error. Extensive experiments on representative multi-task benchmarks demonstrate that SIMS consistently outperforms existing scalarization methods and achieves state-of-the-art performance.
ER -