Query-efficient model evaluation using cached responses
Abstract
Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment. For modern evaluation frameworks, generating and evaluating a response for all queries can be prohibitively expensive. In practice, responses from previously-evaluated models are often cached -- creating a potential opportunity to use this additional information to decrease the number of queries required to accurately evaluate a new model. In this paper, we introduce an approach for predicting benchmark performance that leverages cached model responses based on the Data Kernel Perspective Space (DKPS), a method for quantifying the relationship between models in the black-box setting. Theoretically, we show that DKPS-based methods are query-efficient under certain conditions. Empirically, we demonstrate that DKPS-based methods achieve the same mean absolute error as baselines with a substantially decreased query budget. We conclude by proposing an offline method for selecting a set of queries that maximizes the goodness-of-fit on reference models, improving prediction accuracy over random query selection.
Related
- Benchmark Prediction with Sparse and Unpaired Cached Responses
- Query-Efficient Rejection Sampling for Parallel Masked Diffusion
- Breaking the Shot-Noise Barrier: Cached Recycled Variance-Reduced Gradients for Quantum Optimization
- Efficient Numerical Evaluation of Arbitrary High-order Partial Derivatives via Algebraic Decomposition
- Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices
BibTeX
@misc{helm2026queryefficient,
title = {{Query-efficient model evaluation using cached responses}},
author = {Hayden Helm and Ben Johnson and Carey Priebe},
howpublished = {arXiv},
year = {2026},
month = jun,
doi = {10.48550/arxiv.2605.07096},
eprint = {2605.07096},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.07096},
}
RIS
TY - UNPB
TI - Query-efficient model evaluation using cached responses
AU - Hayden Helm
AU - Ben Johnson
AU - Carey Priebe
PY - 2026
DA - 2026/06/04
PB - arXiv
DO - 10.48550/arxiv.2605.07096
UR - https://arxiv.org/abs/2605.07096
AB - Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment. For modern evaluation frameworks, generating and evaluating a response for all queries can be prohibitively expensive. In practice, responses from previously-evaluated models are often cached -- creating a potential opportunity to use this additional information to decrease the number of queries required to accurately evaluate a new model. In this paper, we introduce an approach for predicting benchmark performance that leverages cached model responses based on the Data Kernel Perspective Space (DKPS), a method for quantifying the relationship between models in the black-box setting. Theoretically, we show that DKPS-based methods are query-efficient under certain conditions. Empirically, we demonstrate that DKPS-based methods achieve the same mean absolute error as baselines with a substantially decreased query budget. We conclude by proposing an offline method for selecting a set of queries that maximizes the goodness-of-fit on reference models, improving prediction accuracy over random query selection.
ER -