Federated Language Models Under Bandwidth Budgets: Distillation Rates and Conformal Coverage
Abstract
Hospitals, companies, and research consortia want language models that learn from, and answer questions over, data that must stay where it is held, over links with limited bandwidth, such as the satellite links of remote clinics and ships. Averaging the sites' model parameters makes each message grow with the model and requires one shared architecture; methods that exchange predictions avoid both limits, but, to our knowledge, none bounds what compressing those predictions costs. We introduce two methods that compress every message at a small and predictable cost. Federated Probe-Logit Distillation (FPLD) trains a shared language model: each site sends compressed logits on shared unlabeled probes, and a hub distills one model from their average. Federated Conformal Retrieval-Augmented Generation (FC-RAG) answers questions: each site scores candidate answers using its own documents and sends compressed scores, and the hub returns a conformal prediction set that contains the correct answer at a guaranteed rate. Both use one codec, which centers each message and rounds it after adding a random offset known to the receiver. For logits within a fixed clipping range, we bound the expected log-loss added by compression independently of vocabulary size; the bound depends only on the bits, the range, and the number of sites, falls as more sites take part, and lets the bit budget be set before deployment. We also prove that recalibrating after compression keeps the coverage guarantee, and that FPLD's training error bounds how far coverage can slip when a threshold is reused after a model update. In experiments with GPT-2 and Qwen2.5, 8 bits per number cause no detectable loss in the trained model or in coverage. FPLD also trains a shared model across different architectures, where parameter averaging does not apply; with one shared architecture, parameter averaging remains more accurate.
Related
- HintKD: Hint Knowledge Distillation for Bandwidth-Constrained Cloud-Edge Inference
- FedCap-Priv: Geometry-Aware Privacy and Bandwidth-Efficient Routing for Federated Capsule Learning
- Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
- Conformal Prediction and Conditional Coverage for Tabular Foundation Models
- Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection
BibTeX
@misc{dubey2026federated,
title = {{Federated Language Models Under Bandwidth Budgets: Distillation Rates and Conformal Coverage}},
author = {Prasanjit Dubey and Xiaoming Huo},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.09986},
eprint = {2605.09986},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.09986},
}
RIS
TY - UNPB
TI - Federated Language Models Under Bandwidth Budgets: Distillation Rates and Conformal Coverage
AU - Prasanjit Dubey
AU - Xiaoming Huo
PY - 2026
DA - 2026/10/09
PB - arXiv
DO - 10.48550/arxiv.2605.09986
UR - https://arxiv.org/abs/2605.09986
AB - Hospitals, companies, and research consortia want language models that learn from, and answer questions over, data that must stay where it is held, over links with limited bandwidth, such as the satellite links of remote clinics and ships. Averaging the sites' model parameters makes each message grow with the model and requires one shared architecture; methods that exchange predictions avoid both limits, but, to our knowledge, none bounds what compressing those predictions costs. We introduce two methods that compress every message at a small and predictable cost. Federated Probe-Logit Distillation (FPLD) trains a shared language model: each site sends compressed logits on shared unlabeled probes, and a hub distills one model from their average. Federated Conformal Retrieval-Augmented Generation (FC-RAG) answers questions: each site scores candidate answers using its own documents and sends compressed scores, and the hub returns a conformal prediction set that contains the correct answer at a guaranteed rate. Both use one codec, which centers each message and rounds it after adding a random offset known to the receiver. For logits within a fixed clipping range, we bound the expected log-loss added by compression independently of vocabulary size; the bound depends only on the bits, the range, and the number of sites, falls as more sites take part, and lets the bit budget be set before deployment. We also prove that recalibrating after compression keeps the coverage guarantee, and that FPLD's training error bounds how far coverage can slip when a threshold is reused after a model update. In experiments with GPT-2 and Qwen2.5, 8 bits per number cause no detectable loss in the trained model or in coverage. FPLD also trains a shared model across different architectures, where parameter averaging does not apply; with one shared architecture, parameter averaging remains more accurate.
ER -