Distributed Grid Search for Hyperparameter Tuning
Abstract
The performance of many distributed optimization algorithms over peer-to-peer networks strongly depends on the choice of their hyperparameters. However, existing selection rules often require global quantities, yield conservative choices, or are tied to specific algorithms and hyperparameters. We propose Distributed Grid Search (D-GS), a consensus-based search method that evaluates and selects hyperparameter configurations across the network, requiring only the ability to run the tuned algorithm and evaluate local gradients. We also propose a zoom-in variant, Distributed Zoom Grid Search (D-ZGS), which improves empirical performance at a reduced cost. We prove that both methods terminate in finite time with all agents holding the same hyperparameter configuration, and quantify the (sub)optimality of this configuration. Numerical experiments on linear and logistic regression show that D-GS closely matches a centralized reference and that tuning and deployment cost a fraction of deploying algorithms with hyperparameters tuned using theoretical results.
Related
- Statistically Valid Post-Training Hyperparameter Selection: From Tuning to Guarantees
- HyperMC: Multi-Fidelity Hyperparameter Tuning for Stochastic Gradient MCMC
- The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation.
- Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
- Towards Hyperparameter Transfer for Differentially Private Optimization
BibTeX
@misc{cachim2026distributed,
title = {{Distributed Grid Search for Hyperparameter Tuning}},
author = {Constança De Herdeiro Cachim and Andrea Martin and Nicola Bastianello},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.41537},
eprint = {2610.41537},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.41537},
}
RIS
TY - UNPB
TI - Distributed Grid Search for Hyperparameter Tuning
AU - Constança De Herdeiro Cachim
AU - Andrea Martin
AU - Nicola Bastianello
PY - 2026
DA - 2026/10/30
PB - arXiv
DO - 10.48550/arxiv.2610.41537
UR - https://arxiv.org/abs/2610.41537
AB - The performance of many distributed optimization algorithms over peer-to-peer networks strongly depends on the choice of their hyperparameters. However, existing selection rules often require global quantities, yield conservative choices, or are tied to specific algorithms and hyperparameters. We propose Distributed Grid Search (D-GS), a consensus-based search method that evaluates and selects hyperparameter configurations across the network, requiring only the ability to run the tuned algorithm and evaluate local gradients. We also propose a zoom-in variant, Distributed Zoom Grid Search (D-ZGS), which improves empirical performance at a reduced cost. We prove that both methods terminate in finite time with all agents holding the same hyperparameter configuration, and quantify the (sub)optimality of this configuration. Numerical experiments on linear and logistic regression show that D-GS closely matches a centralized reference and that tuning and deployment cost a fraction of deploying algorithms with hyperparameters tuned using theoretical results.
ER -