Federated learning with heavy-tailed gradient noise and communication noise: a variance-reduction based algorithm
Abstract
Federated learning (FL) is an emerging distributed machine learning paradigm that enables local devices to jointly train a global model while keeping data decentralized and private. We propose a variance-reduction based algorithm, VRA-FedSGD, for FL in the presence of heavy-tailed gradient noise and communication noise, where these noises are prevalent in large-scale machine learning over wireless networks and Internet of Things deployments. VRA-FedSGD employs a momentum variance reduction technique together with a nonlinear mapping to mitigate heavy-tailed gradient noise, and uses a variance-reduced aggregation mechanism to suppress heavy-tailed communication noise. In the mean sense, VRA-FedSGD achieves a convergence rate of {\small𝒪(K−(p−1)/(2p−1))} for nonconvex objective functions, where p is the tail index of heavy-tailed noise. In the almost sure sense, VRA-FedSGD achieves a convergence rate of 𝒪̃(K−(1−1/(p−ε))) for strongly convex objective functions, where ε is an arbitrarily small constant. Simulated experiments on a logistic regression problem with real-world data verify the effectiveness of VRA-FedSGD.
Related
- Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Cost
- FedRSPO+: A Heterogeneity-aware Algorithm for Decision-focused Federated Learning
- Uncertainty-Aware Federated Learning for Joint Communication and Computation in XR Systems
- FedPGT: Progressive Gradient Transmission for Vehicular Federated Learning over Time-Varying Channels
- A Particle-Swarm-Assisted Gradient Meta-Learning Algorithm for Joint Transmit Precoding and STAR-RIS Coefficient Optimization
BibTeX
@misc{zhao2026federated,
title = {{Federated learning with heavy-tailed gradient noise and communication noise: a variance-reduction based algorithm}},
author = {Shengchao Zhao and Yongchao Liu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2606.22466},
eprint = {2606.22466},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.22466},
}
RIS
TY - UNPB
TI - Federated learning with heavy-tailed gradient noise and communication noise: a variance-reduction based algorithm
AU - Shengchao Zhao
AU - Yongchao Liu
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2606.22466
UR - https://arxiv.org/abs/2606.22466
AB - Federated learning (FL) is an emerging distributed machine learning paradigm that enables local devices to jointly train a global model while keeping data decentralized and private. We propose a variance-reduction based algorithm, VRA-FedSGD, for FL in the presence of heavy-tailed gradient noise and communication noise, where these noises are prevalent in large-scale machine learning over wireless networks and Internet of Things deployments. VRA-FedSGD employs a momentum variance reduction technique together with a nonlinear mapping to mitigate heavy-tailed gradient noise, and uses a variance-reduced aggregation mechanism to suppress heavy-tailed communication noise. In the mean sense, VRA-FedSGD achieves a convergence rate of {\small𝒪(K^−(p−1)/(2p−1))} for nonconvex objective functions, where p is the tail index of heavy-tailed noise. In the almost sure sense, VRA-FedSGD achieves a convergence rate of 𝒪̃(K^−(1−1/(p−ε))) for strongly convex objective functions, where ε is an arbitrarily small constant. Simulated experiments on a logistic regression problem with real-world data verify the effectiveness of VRA-FedSGD.
ER -