Server-Side Sparse Assignment for Federated Mixture-of-Experts
Abstract
Large language models (LLMs) are increasingly recognized for their remarkable capabilities; federated learning (FL) offers a promising framework for collaboratively fine-tuning LLMs on edge devices while achieving data privacy. However, edge devices are inherently heterogeneous: clients may exhibit diverse data distributions and resource constraints, including limited memory and computation budgets. Recent federated Mixture-of-Experts (MoE) approaches address these challenges by using sparse expert activation or partial expert loading. However, existing partial-expert methods rely on heuristic full-pool observations or prioritize system-wide load balance, without guarantees for recovering client-specific relevant experts from partial observations. We propose S3-MoE, a server-side sparse expert assignment framework that estimates client-specific expert relevance from partial observations and refines assignments through an exploit--explore mechanism. We also establish relevance-estimation error and exact Top-K recovery guarantees. Experiments using a CNN-based MoE on CIFAR-100 show that S3-MoE recovers task-dependent client--expert preferences and reduces conflicts caused by correlated expert selection across clients. Compared with Full-20 FedAvg, which gives each client access to all 20 experts for local Top-5 routing, S3-MoE achieves 1.31 percentage points higher mean final accuracy while loading only $5$ experts per client.
Related
- When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts
- Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts
- Relative-Error Composite Federated Optimization with One Server-Side Proximal Step per Round
- Mixture of Censored Partially Linear Experts Models for Symmetric or Asymmetric Data with or without Outlying Observations: A novel approach with two-piece scale-mixture of normal distributions
- A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization
BibTeX
@misc{zheng2026serverside,
title = {{Server-Side Sparse Assignment for Federated Mixture-of-Experts}},
author = {Xue Zheng and Aylin Yener},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.07093},
eprint = {2611.07093},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.07093},
}
RIS
TY - UNPB
TI - Server-Side Sparse Assignment for Federated Mixture-of-Experts
AU - Xue Zheng
AU - Aylin Yener
PY - 2026
DA - 2026/11/04
PB - arXiv
DO - 10.48550/arxiv.2611.07093
UR - https://arxiv.org/abs/2611.07093
AB - Large language models (LLMs) are increasingly recognized for their remarkable capabilities; federated learning (FL) offers a promising framework for collaboratively fine-tuning LLMs on edge devices while achieving data privacy. However, edge devices are inherently heterogeneous: clients may exhibit diverse data distributions and resource constraints, including limited memory and computation budgets. Recent federated Mixture-of-Experts (MoE) approaches address these challenges by using sparse expert activation or partial expert loading. However, existing partial-expert methods rely on heuristic full-pool observations or prioritize system-wide load balance, without guarantees for recovering client-specific relevant experts from partial observations. We propose S^3-MoE, a server-side sparse expert assignment framework that estimates client-specific expert relevance from partial observations and refines assignments through an exploit--explore mechanism. We also establish relevance-estimation error and exact Top-K recovery guarantees. Experiments using a CNN-based MoE on CIFAR-100 show that S^3-MoE recovers task-dependent client--expert preferences and reduces conflicts caused by correlated expert selection across clients. Compared with Full-20 FedAvg, which gives each client access to all 20 experts for local Top-5 routing, S^3-MoE achieves 1.31 percentage points higher mean final accuracy while loading only $5$ experts per client.
ER -