Speculative Evaluation of Stochastic LLMs
Abstract
Evaluating a stochastic large language model is costly: benchmark scores estimate expected performance from randomized rollouts, yet uniform repetition ignores sharp differences in task-level rollout variance. We ask how to minimize the variance of a fixed-benchmark mean under an exact rollout budget. We develop Speculative Evaluation with a Hierarchical Bayesian Neyman (HBN) policy with pilot size and stage weight jointly chosen ex ante. It runs a short uniform pilot, pools per-task success counts with a hierarchical Bayesian model, and uses posterior expectations of task-level sampling variances for exact positive-integer Neyman allocation. To mitigate the pilot synchronization barrier, HBN-async speculatively executes continuations from partial pilot feedback and retains those selected by the final allocation. Across six checkpoints and 18 benchmark groups, we evaluate 107 nondegenerate benchmark-checkpoint profiles. For rollout budgets of 8-64 per task, Speculative Evaluation reduces variance relative to Uniform by 12.8%-33.6% on average across profiles, outperforming hindsight-tuned empirical and independent Bayesian baselines. Real-generation experiments that account for the pilot synchronization barrier show that HBN-async mitigates its overhead, helping translate statistical efficiency into practical evaluation benefits.
Related
- The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts
- Explore Once, Evaluate Many: Multiple Policy Evaluation for Stochastic Linear Bandits
- Accelerating Diffusion Sampling via Speculative Draft Trees
- Multiple Importance Sampling for Guided Speculative Inference
- Distributed LLM Inference via Communication-Efficient Speculative Decoding
BibTeX
@misc{shen2026speculative,
title = {{Speculative Evaluation of Stochastic LLMs}},
author = {Qianli Shen and Xiang Li and Ruomeng Ding and Yanxi Chen and Daoyuan Chen and Yaliang Li},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.28560},
eprint = {2609.28560},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.28560},
}
RIS
TY - UNPB
TI - Speculative Evaluation of Stochastic LLMs
AU - Qianli Shen
AU - Xiang Li
AU - Ruomeng Ding
AU - Yanxi Chen
AU - Daoyuan Chen
AU - Yaliang Li
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.28560
UR - https://arxiv.org/abs/2609.28560
AB - Evaluating a stochastic large language model is costly: benchmark scores estimate expected performance from randomized rollouts, yet uniform repetition ignores sharp differences in task-level rollout variance. We ask how to minimize the variance of a fixed-benchmark mean under an exact rollout budget. We develop Speculative Evaluation with a Hierarchical Bayesian Neyman (HBN) policy with pilot size and stage weight jointly chosen ex ante. It runs a short uniform pilot, pools per-task success counts with a hierarchical Bayesian model, and uses posterior expectations of task-level sampling variances for exact positive-integer Neyman allocation. To mitigate the pilot synchronization barrier, HBN-async speculatively executes continuations from partial pilot feedback and retains those selected by the final allocation. Across six checkpoints and 18 benchmark groups, we evaluate 107 nondegenerate benchmark-checkpoint profiles. For rollout budgets of 8-64 per task, Speculative Evaluation reduces variance relative to Uniform by 12.8%-33.6% on average across profiles, outperforming hindsight-tuned empirical and independent Bayesian baselines. Real-generation experiments that account for the pilot synchronization barrier show that HBN-async mitigates its overhead, helping translate statistical efficiency into practical evaluation benefits.
ER -