Benchmark Prediction with Sparse and Unpaired Cached Responses
Abstract
Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired. In this paper, we introduce the Product Kernel Perspective Space (PKPS), a collection of low-dimensional representations of models based on their embedded responses. PKPS generalizes existing methods for representing black-box generative models by comparing responses to similar -- rather than just identical -- queries and is therefore able to utilize sparse and unpaired information. Theoretically, we show that benchmark prediction using PKPS is query-efficient relative to methods that only use information from dense and paired portions of the evaluation data. Empirically, we show that PKPS-based methods outperform baselines on benchmark prediction across two large benchmark suites.
Related
- Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark
- Sparse Latent Class Analysis For Dichotomous Responses: Post-Estimation Refinement via Item-level Pseudo-Likelihood
- Towards AI-Driven Nanomedicine Discovery: A Benchmark and Multimodal Learning Framework for Nano Self-Assembly Prediction
- Breaking the Shot-Noise Barrier: Cached Recycled Variance-Reduced Gradients for Quantum Optimization
- DisKO: Deep Koopman Learning in Distribution Space from Unpaired Snapshots
BibTeX
@misc{helm2026benchmark,
title = {{Benchmark Prediction with Sparse and Unpaired Cached Responses}},
author = {Hayden Helm and Avijit Ghosh and Ben Johnson and Carey Priebe},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.03614},
eprint = {2611.03614},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.03614},
}
RIS
TY - UNPB
TI - Benchmark Prediction with Sparse and Unpaired Cached Responses
AU - Hayden Helm
AU - Avijit Ghosh
AU - Ben Johnson
AU - Carey Priebe
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.03614
UR - https://arxiv.org/abs/2611.03614
AB - Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired. In this paper, we introduce the Product Kernel Perspective Space (PKPS), a collection of low-dimensional representations of models based on their embedded responses. PKPS generalizes existing methods for representing black-box generative models by comparing responses to similar -- rather than just identical -- queries and is therefore able to utilize sparse and unpaired information. Theoretically, we show that benchmark prediction using PKPS is query-efficient relative to methods that only use information from dense and paired portions of the evaluation data. Empirically, we show that PKPS-based methods outperform baselines on benchmark prediction across two large benchmark suites.
ER -