Beyond Conventional Federated Learning via High-Order Regularization
Abstract
Federated clients that perform several local optimization steps can return parameter displacements with widely different magnitudes. The quadratic regularization of FedProx grows linearly with displacement and therefore offers limited control over the contrast between ordinary and unusually large client movements. We here introduce HiFedProx, which replaces the quadratic penalty with a scale-matched power-type regularizer indexed by p≥2. All powers have the same regularization-gradient magnitude at a reference displacement R, while every p>2 gives a weaker response below R and a stronger response above it. An exact affine reference calculation shows that increasing p compresses relative displacement disparities, although very large powers approach fixed-radius behavior and increase local curvature. HiFedProx combines this geometry with finite-budget stochastic client optimization and same-minibatch Armijo backtracking. In paired five-seed experiments on a frozen 60-writer FEMNIST subset, a common-parameter study over p∈{2,3,4,5,6,7,8} shows similar clean-training performance but substantial gains under composite stress. The lowest moderate- and severe-stress losses occur at p=7 and p=6, improving over p=2 by 11.44% and 23.16%, respectively. Although displacement-tail ratios continue to decrease through p=8, predictive performance peaks in an intermediate range and Armijo trial cost increases with p. These results indicate that the exponent should be calibrated rather than maximized. In our experiments, p=5--$7$ provides the most useful range.
Related
- Resilience Beyond Stationary Client Unavailability: Unlocking Efficient and Unbiased Federated Learning
- Beyond Aggregate Drift: Optimal Client-Specific Windows in Federated Learning
- Breaking the Odd Cycle: Regularization and First-Order Optimization for the Maximum s-Plex Problem
- Multi-rate high-order regularization with an order-(p−2) oracle
- Fractional-order differential geometry of Gielis curves: chart regularization, Caputo curvature, and the fate of singularities
BibTeX
@misc{kabgani2026beyond,
title = {{Beyond Conventional Federated Learning via High-Order Regularization}},
author = {Alireza Kabgani and Masoud Ahookhosh},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.09904},
eprint = {2609.09904},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.09904},
}
RIS
TY - UNPB
TI - Beyond Conventional Federated Learning via High-Order Regularization
AU - Alireza Kabgani
AU - Masoud Ahookhosh
PY - 2026
DA - 2026/09/09
PB - arXiv
DO - 10.48550/arxiv.2609.09904
UR - https://arxiv.org/abs/2609.09904
AB - Federated clients that perform several local optimization steps can return parameter displacements with widely different magnitudes. The quadratic regularization of FedProx grows linearly with displacement and therefore offers limited control over the contrast between ordinary and unusually large client movements. We here introduce HiFedProx, which replaces the quadratic penalty with a scale-matched power-type regularizer indexed by p≥2. All powers have the same regularization-gradient magnitude at a reference displacement R, while every p>2 gives a weaker response below R and a stronger response above it. An exact affine reference calculation shows that increasing p compresses relative displacement disparities, although very large powers approach fixed-radius behavior and increase local curvature. HiFedProx combines this geometry with finite-budget stochastic client optimization and same-minibatch Armijo backtracking. In paired five-seed experiments on a frozen 60-writer FEMNIST subset, a common-parameter study over p∈{2,3,4,5,6,7,8} shows similar clean-training performance but substantial gains under composite stress. The lowest moderate- and severe-stress losses occur at p=7 and p=6, improving over p=2 by 11.44% and 23.16%, respectively. Although displacement-tail ratios continue to decrease through p=8, predictive performance peaks in an intermediate range and Armijo trial cost increases with p. These results indicate that the exponent should be calibrated rather than maximized. In our experiments, p=5--$7$ provides the most useful range.
ER -