Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons
Abstract
This paper considers a decision-maker tasked with identifying the k best items from a large pool of candidates, such as job applicants, LLMs, award nominees, or new product ideas. The decision-maker learns about the items' quality by soliciting human evaluations. Each evaluation can be costly, so the decision-maker aims to use the limited evaluation budget efficiently by allocating evaluations adaptively. A key consideration is the form of evaluation to solicit, which should be something evaluators are comfortable providing. Two popular forms are ratings and pairwise comparisons, and we allow for both, as they may be especially useful in combination given the limited budget. Ratings can be used early on to get a coarse picture of which items are good and which are not. As that picture forms, comparisons can refine the ordering of the good items, narrowing in on the ultimate top-k selection. To solve the allocation problem, we develop an adaptive algorithm built from two allocation policies. The first, our primary contribution, is based on a surrogate objective that is naturally forward-looking, valuing evaluations that help inform what to evaluate in future rounds, not just those that directly improve the selection decision. The decision rule is interpretable, and we prove that it is asymptotically consistent. Late in the evaluation process, when being forward-looking is less productive, we construct a myopic policy and a criterion for switching to it. In simulation, the method gives insight into how to combine these two types of evaluations, and beats the (limited) existing benchmarks.
Related
- Adaptively Combining Randomized and External Control Data Using a Mixture Prior in the Presence of Heterogeneity
- Selecting the 7th Smallest of 16 Elements Takes Exactly 28 Comparisons
- Pairwise comparisons meet optimal transport theory
- How Many Repeated Pairwise Comparisons Are Needed for Ranking under Heterogeneity?
- Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons
BibTeX
@misc{goldberger2026selecting,
title = {{Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons}},
author = {Motti Goldberger and Nils Rudi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.26380},
eprint = {2610.26380},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.26380},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons
AU - Motti Goldberger
AU - Nils Rudi
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.26380
UR - https://arxiv.org/abs/2610.26380
AB - This paper considers a decision-maker tasked with identifying the k best items from a large pool of candidates, such as job applicants, LLMs, award nominees, or new product ideas. The decision-maker learns about the items' quality by soliciting human evaluations. Each evaluation can be costly, so the decision-maker aims to use the limited evaluation budget efficiently by allocating evaluations adaptively. A key consideration is the form of evaluation to solicit, which should be something evaluators are comfortable providing. Two popular forms are ratings and pairwise comparisons, and we allow for both, as they may be especially useful in combination given the limited budget. Ratings can be used early on to get a coarse picture of which items are good and which are not. As that picture forms, comparisons can refine the ordering of the good items, narrowing in on the ultimate top-k selection. To solve the allocation problem, we develop an adaptive algorithm built from two allocation policies. The first, our primary contribution, is based on a surrogate objective that is naturally forward-looking, valuing evaluations that help inform what to evaluate in future rounds, not just those that directly improve the selection decision. The decision rule is interpretable, and we prove that it is asymptotically consistent. Late in the evaluation process, when being forward-looking is less productive, we construct a myopic policy and a criterion for switching to it. In simulation, the method gives insight into how to combine these two types of evaluations, and beats the (limited) existing benchmarks.
ER -