Statistically Valid Post-Training Hyperparameter Selection: From Tuning to Guarantees
Abstract
Post-training hyperparameter selection is a critical step in the deployment of modern artificial intelligence systems, given the need to tune degrees of freedom of pre-trained models such as inference-time parameters, implementation-level settings, and thresholds driving decision rules. Despite its practical importance, hyperparameter selection is typically performed using best-effort empirical methods such as grid search or Bayesian optimization, which provide no formal statistical guarantees on reliability or safety. This monograph, intended for an audience of signal processing and machine learning researchers, presents a unified statistical framework for reliable post-training hyperparameter selection, centered on the learn-then-test (LTT) paradigm. LTT formulates the hyperparameter selection problem as multiple hypothesis testing over a candidate set of hyperparameters. The framework enables the choice of hyperparameters that provably satisfy application-specific reliability requirements---such as bounds on average risk, quantile risk, or information-theoretic constraints---with explicit, finite-sample control of error probabilities. The supporting statistical machinery, namely p-values, e-values, and concentration inequalities, is developed from first principles.
Related
- Distributed Grid Search for Hyperparameter Tuning
- HyperMC: Multi-Fidelity Hyperparameter Tuning for Stochastic Gradient MCMC
- The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation.
- Valid Data‐Driven Subgroup Selection With Uniform Guarantees in Clinical Trials
- Learning in Inverse Games: Tractable Training with Probabilistic Guarantees
BibTeX
@misc{farzaneh2026statistically,
title = {{Statistically Valid Post-Training Hyperparameter Selection: From Tuning to Guarantees}},
author = {Amirmohammad Farzaneh and Osvaldo Simeone},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.25601},
eprint = {2606.25601},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.25601},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Statistically Valid Post-Training Hyperparameter Selection: From Tuning to Guarantees
AU - Amirmohammad Farzaneh
AU - Osvaldo Simeone
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2606.25601
UR - https://arxiv.org/abs/2606.25601
AB - Post-training hyperparameter selection is a critical step in the deployment of modern artificial intelligence systems, given the need to tune degrees of freedom of pre-trained models such as inference-time parameters, implementation-level settings, and thresholds driving decision rules. Despite its practical importance, hyperparameter selection is typically performed using best-effort empirical methods such as grid search or Bayesian optimization, which provide no formal statistical guarantees on reliability or safety. This monograph, intended for an audience of signal processing and machine learning researchers, presents a unified statistical framework for reliable post-training hyperparameter selection, centered on the learn-then-test (LTT) paradigm. LTT formulates the hyperparameter selection problem as multiple hypothesis testing over a candidate set of hyperparameters. The framework enables the choice of hyperparameters that provably satisfy application-specific reliability requirements---such as bounds on average risk, quantile risk, or information-theoretic constraints---with explicit, finite-sample control of error probabilities. The supporting statistical machinery, namely p-values, e-values, and concentration inequalities, is developed from first principles.
ER -