Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory
Abstract
Test-time scaling can improve the performance of large language models by generating multiple answers to a question and aggregating them. Weighted ensembles extend this idea by drawing answers from several models, with the weights determining how often each model is used. Learning these weights from a training dataset involves two sample sizes: Q labeled questions and N generations per model and question. We study their interplay through the clean risk, the population error of the ensemble's infinite-generation voting rule. Under a margin condition with exponent β, we show that empirical risk minimization (ERM) with a calibrated margin buffer achieves excess risk of order Q−1/2+N−β/2, up to logarithmic factors, for fixed numbers of models and candidate answers. To investigate the optimality of this dependence on Q and N, we analyze a simplified submodel with two models and two possible answers per question. We characterize its minimax excess risk up to constants by a Hellinger modulus for a binomial-mixture experiment. Polynomial approximation and spectral analysis reveal three regimes. When questions are relatively scarce, the Q−1/2 term governs learning and the general buffered ERM is optimal. In an intermediate regime, pooling generation counts across questions improves upon estimating each answer distribution separately, with only logarithmic gains from additional questions. With sufficiently many questions, the minimax excess risk reaches an N−β identification floor that persists even with infinitely many questions. The submodel's lower bounds also apply to general ensembles, matching the buffered-ERM upper bound up to logarithmic factors when questions are scarce and in parts of the intermediate regime. Questions remain about the transitions between regimes and the construction of matching pooled estimators for general ensembles.
Related
- A statistical theory for blind denoising: minimax estimation of the noise level from a single sample
- Non-Adaptive 1-Bit Mean Estimation: Minimax Rates and the Sample-Interval Tradeoff
- Near-Minimax Sample Complexity for Multi-Objective Reinforcement Learning under Generative Models
- Robust Local Fréchet Regression: Nonasymptotic Theory and Minimax Optimality
- Statistical Inference for Persistence Diagrams via Landmark Embeddings: Minimax Theory and Finite Approximation
BibTeX
@misc{zhang2026sample,
title = {{Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory}},
author = {Zhi Zhang and Arash A. Amini},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.21363},
eprint = {2610.21363},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.21363},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory
AU - Zhi Zhang
AU - Arash A. Amini
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.21363
UR - https://arxiv.org/abs/2610.21363
AB - Test-time scaling can improve the performance of large language models by generating multiple answers to a question and aggregating them. Weighted ensembles extend this idea by drawing answers from several models, with the weights determining how often each model is used. Learning these weights from a training dataset involves two sample sizes: Q labeled questions and N generations per model and question. We study their interplay through the clean risk, the population error of the ensemble's infinite-generation voting rule. Under a margin condition with exponent β, we show that empirical risk minimization (ERM) with a calibrated margin buffer achieves excess risk of order Q^−1/2+N^−β/2, up to logarithmic factors, for fixed numbers of models and candidate answers. To investigate the optimality of this dependence on Q and N, we analyze a simplified submodel with two models and two possible answers per question. We characterize its minimax excess risk up to constants by a Hellinger modulus for a binomial-mixture experiment. Polynomial approximation and spectral analysis reveal three regimes. When questions are relatively scarce, the Q^−1/2 term governs learning and the general buffered ERM is optimal. In an intermediate regime, pooling generation counts across questions improves upon estimating each answer distribution separately, with only logarithmic gains from additional questions. With sufficiently many questions, the minimax excess risk reaches an N^−β identification floor that persists even with infinitely many questions. The submodel's lower bounds also apply to general ensembles, matching the buffered-ERM upper bound up to logarithmic factors when questions are scarce and in parts of the intermediate regime. Questions remain about the transitions between regimes and the construction of matching pooled estimators for general ensembles.
ER -