Byzantine-Robust Federated Representation Learning
Abstract
We study federated learning (FL) with adversarial clients, where the goal is to minimize the average loss of the honest (non-adversarial) clients without knowing their identity. Under heterogeneity, a single shared model parameter is statistically inappropriate: it cannot capture the distinct data-generating processes across clients, incurring an irreducible model-heterogeneity bias and severely limiting robustness to adversarial clients (a.k.a. Byzantine-robustness). We address this problem through representation learning, where each client learns a personalized linear head, while collaboratively estimating a shared nonlinear representation through Byzantine-robust aggregation. We demonstrate that the heterogeneity among honest representation gradients is controlled by the representation error and statistical errors that decay either with the number of data samples per client (τ) or the number of iterations (T). In particular, our non-asymptotic parameter recovery error bound reveals three terms: (i) an initialization-dependent error that goes away with T, (ii) finite-sample noise terms that decreases with τ and the number of honest clients, and (iii) a stochastic gradient variance term that also reduces with T. Importantly, with no irreducible model-heterogeneity bias in our bounds. We extend the regression analysis to multiclass classification, and empirically validate it on CIFAR-10, FEMNIST, and School Exam Score datasets.
Related
- Byzantine-Robust Federated RAG via Aligned Calibration and Fixed-Membership Conformal Prediction
- Distributionally Robust Federated Learning with Multi-Source Data
- Robust and Sparse Personalized Federated Learning via Gamma-Divergence
- Byzantine-Robust Sparse Regression with Missing Covariates
- Heterogeneity-calibrated Byzantine-robust distributed composite quantile regression
BibTeX
@misc{toso2026byzantinerobust,
title = {{Byzantine-Robust Federated Representation Learning}},
author = {Leonardo F. Toso and James Anderson and Rafael Pinot and Nirupam Gupta},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.36660},
eprint = {2609.36660},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.36660},
}
RIS
TY - UNPB
TI - Byzantine-Robust Federated Representation Learning
AU - Leonardo F. Toso
AU - James Anderson
AU - Rafael Pinot
AU - Nirupam Gupta
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.36660
UR - https://arxiv.org/abs/2609.36660
AB - We study federated learning (FL) with adversarial clients, where the goal is to minimize the average loss of the honest (non-adversarial) clients without knowing their identity. Under heterogeneity, a single shared model parameter is statistically inappropriate: it cannot capture the distinct data-generating processes across clients, incurring an irreducible model-heterogeneity bias and severely limiting robustness to adversarial clients (a.k.a. Byzantine-robustness). We address this problem through representation learning, where each client learns a personalized linear head, while collaboratively estimating a shared nonlinear representation through Byzantine-robust aggregation. We demonstrate that the heterogeneity among honest representation gradients is controlled by the representation error and statistical errors that decay either with the number of data samples per client (τ) or the number of iterations (T). In particular, our non-asymptotic parameter recovery error bound reveals three terms: (i) an initialization-dependent error that goes away with T, (ii) finite-sample noise terms that decreases with τ and the number of honest clients, and (iii) a stochastic gradient variance term that also reduces with T. Importantly, with no irreducible model-heterogeneity bias in our bounds. We extend the regression analysis to multiclass classification, and empirically validate it on CIFAR-10, FEMNIST, and School Exam Score datasets.
ER -