Robustifying Asynchronous SGD via Soft Throttling
Abstract
Asynchronous SGD is a popular algorithm for distributed learning where each client's gradient update is applied on arrival. This leads to a speed-up, but also an increased vulnerability to attacks, as fast clients can dominate the total update. We introduce Throttle, a Byzantine-robust generalization of asynchronous SGD where the key idea is to exponentially down-weight updates from faster clients by a factor q. Both asynchronous SGD (q=1) and synchronous Byzantine-robust SGD (q→∞) correspond to specific settings of Throttle. We provide a theoretical analysis of the convergence rate and validate the robustness to attacks both theoretically and empirically. Remarkably, our experiments show that this down-weighting mechanism can also improve performance over standard asynchronous SGD even in the non-Byzantine setting.
Related
- A Spike-Driven Neuromorphic Framework for Asynchronous Population-Based Optimisation
- L2-error bounds for covariation estimation under asynchronous mixed time-space sampling
- Asynchronous Sensitivity-Based Distributed NMPC
- Asynchronous Parallel Computation of Lagrangian Bounds for Mixed-Integer Stochastic Programs by Frank-Wolfe Methods within Projective Hedging
- An Approach to Asynchronous Unsourced Random Access
BibTeX
@misc{otsuka2026robustifying,
title = {{Robustifying Asynchronous SGD via Soft Throttling}},
author = {Kaoru Otsuka and Maxime Meyer and Yuki Takezawa and Makoto Yamada and Anastasia Koloskova},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.39357},
eprint = {2609.39357},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.39357},
}
RIS
TY - UNPB
TI - Robustifying Asynchronous SGD via Soft Throttling
AU - Kaoru Otsuka
AU - Maxime Meyer
AU - Yuki Takezawa
AU - Makoto Yamada
AU - Anastasia Koloskova
PY - 2026
DA - 2026/09/30
PB - arXiv
DO - 10.48550/arxiv.2609.39357
UR - https://arxiv.org/abs/2609.39357
AB - Asynchronous SGD is a popular algorithm for distributed learning where each client's gradient update is applied on arrival. This leads to a speed-up, but also an increased vulnerability to attacks, as fast clients can dominate the total update. We introduce Throttle, a Byzantine-robust generalization of asynchronous SGD where the key idea is to exponentially down-weight updates from faster clients by a factor q. Both asynchronous SGD (q=1) and synchronous Byzantine-robust SGD (q→∞) correspond to specific settings of Throttle. We provide a theoretical analysis of the convergence rate and validate the robustness to attacks both theoretically and empirically. Remarkably, our experiments show that this down-weighting mechanism can also improve performance over standard asynchronous SGD even in the non-Byzantine setting.
ER -