Distributed LLM Inference via Communication-Efficient Speculative Decoding
Abstract
Distributed speculative decoding has emerged as a promising approach for device--server collaborative large language model (LLM) inference, but its decoding speedup is often limited by the vocabulary-scale communication overhead. In existing distributed speculative decoding methods, exact rejection sampling requires transmitting full token distributions over the LLM vocabulary between the device and the server, which becomes costly in bandwidth-constrained environments. To address this issue, we propose DISCO, a communication-efficient framework for distributed speculative decoding, that eliminates vocabulary-scale distribution transfer while preserving the exact target distribution. DISCO achieves this by introducing a server-side replica of the draft small language model (SLM), which enables the server to locally obtain the draft-side distribution required for rejection sampling. As a result, DISCO exchanges only drafted token indices and scalar probability values between the device and the server. Experimental results on multiple datasets show that DISCO consistently achieves higher decoding speedup than existing distributed speculative decoding methods, particularly under stringent bandwidth constraints.
Related
- Communication-Efficient Differentially Private Gradient Tracking for Distributed Optimization via Local Updates
- Communication-Efficient Distributed Training via Ring-Based Coded Approximate All-Reduce
- Dynamic Regret and Tightened Constraint Violation for Communication-Efficient Distributed Online Bilevel Optimization
- The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts
- Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
BibTeX
@misc{shin2026distributed,
title = {{Distributed LLM Inference via Communication-Efficient Speculative Decoding}},
author = {Joohyun Shin and Byeonghyeon Park and Keumseo Ryum and Joonhyuk Kang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.35957},
eprint = {2610.35957},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.35957},
}
RIS
TY - UNPB
TI - Distributed LLM Inference via Communication-Efficient Speculative Decoding
AU - Joohyun Shin
AU - Byeonghyeon Park
AU - Keumseo Ryum
AU - Joonhyuk Kang
PY - 2026
DA - 2026/10/25
PB - arXiv
DO - 10.48550/arxiv.2610.35957
UR - https://arxiv.org/abs/2610.35957
AB - Distributed speculative decoding has emerged as a promising approach for device--server collaborative large language model (LLM) inference, but its decoding speedup is often limited by the vocabulary-scale communication overhead. In existing distributed speculative decoding methods, exact rejection sampling requires transmitting full token distributions over the LLM vocabulary between the device and the server, which becomes costly in bandwidth-constrained environments. To address this issue, we propose DISCO, a communication-efficient framework for distributed speculative decoding, that eliminates vocabulary-scale distribution transfer while preserving the exact target distribution. DISCO achieves this by introducing a server-side replica of the draft small language model (SLM), which enables the server to locally obtain the draft-side distribution required for rejection sampling. As a result, DISCO exchanges only drafted token indices and scalar probability values between the device and the server. Experimental results on multiple datasets show that DISCO consistently achieves higher decoding speedup than existing distributed speculative decoding methods, particularly under stringent bandwidth constraints.
ER -