ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignment
Abstract
Best-of-n (BoN) sampling is a simple yet effective inference-time alignment method, but hard maximization provides only coarse control over the trade-off between reward and distribution shift. Soft Best-of-n (Verdun et al. 2025) provides smoother control and converges to the optimal distribution associated with KL-regularized reward maximization. In this paper, we introduce ExpBoN, an alternative soft BoN method based on the exponential-noise report-noisy-max mechanism. It admits an exact finite-n decomposition, which yields exponentially fast convergence in total variation, expected reward, and both directions of KL divergence. We provide comprehensive theoretical analyses of its convergence and regret behavior. We further integrate ExpBoN into the guided speculative inference (GSI) framework (Geuter, Mroueh, and AlvarezMelis 2025), resulting in ExpGSI, for efficient reward-guided LLM alignment. ExpGSI yields substantial reductions in computational cost while maintaining comparable accuracy. Experiments on MATH500, MMLU-STEM, and Minerva Math with the Qwen2.5-Math and Qwen3 model families show that ExpGSI reduces estimated computation by 14%-39% across candidate budgets for Qwen2.5-Math and by up to 45% at n=16 for Qwen3. Overall, our results provide a theoretical and algorithmic foundation for exponential-noise BoN and efficient test-time LLM alignment.
Related
- Efficient Best-of-N policy evaluation for inference-time alignment
- Exponential mixing for the stochastic Navier--Stokes equation with localized noise
- Exponential Ergodicity and Invariant Distribution of a Degenerate Reflected SDE with Stable Noise
- Efficient Optimization on the Euclidean Sphere: Riemannian Gradient Alignment as a Unifying Principle
- What masking geometry works best for EEG foundation models?
BibTeX
@misc{liu2026expbon,
title = {{ExpBoN: Exponential-Noise Best-of-$n$ for Efficient Test-Time LLM Alignment}},
author = {Yanxiao Liu and Sicheng Wan and Deniz Gündüz},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.21899},
eprint = {2609.21899},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.21899},
}
RIS
TY - UNPB
TI - ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignment
AU - Yanxiao Liu
AU - Sicheng Wan
AU - Deniz Gündüz
PY - 2026
DA - 2026/09/18
PB - arXiv
DO - 10.48550/arxiv.2609.21899
UR - https://arxiv.org/abs/2609.21899
AB - Best-of-n (BoN) sampling is a simple yet effective inference-time alignment method, but hard maximization provides only coarse control over the trade-off between reward and distribution shift. Soft Best-of-n (Verdun et al. 2025) provides smoother control and converges to the optimal distribution associated with KL-regularized reward maximization. In this paper, we introduce ExpBoN, an alternative soft BoN method based on the exponential-noise report-noisy-max mechanism. It admits an exact finite-n decomposition, which yields exponentially fast convergence in total variation, expected reward, and both directions of KL divergence. We provide comprehensive theoretical analyses of its convergence and regret behavior. We further integrate ExpBoN into the guided speculative inference (GSI) framework (Geuter, Mroueh, and AlvarezMelis 2025), resulting in ExpGSI, for efficient reward-guided LLM alignment. ExpGSI yields substantial reductions in computational cost while maintaining comparable accuracy. Experiments on MATH500, MMLU-STEM, and Minerva Math with the Qwen2.5-Math and Qwen3 model families show that ExpGSI reduces estimated computation by 14%-39% across candidate budgets for Qwen2.5-Math and by up to 45% at n=16 for Qwen3. Overall, our results provide a theoretical and algorithmic foundation for exponential-noise BoN and efficient test-time LLM alignment.
ER -