Efficient Benchmarking Is Just Feature Selection and Multiple Regression
Abstract
Efficient benchmarking techniques aim to lower the computational cost of evaluating LLMs by predicting full benchmark scores using only a subset of a benchmark's questions. By reframing this problem as an instance of multiple regression with feature selection, we find that existing efficient benchmarking methods can be greatly improved by simply using kernel ridge regression at the prediction stage. Additionally, using an information-theoretic feature-selection algorithm called minimum redundancy maximum relevance (mRMR), we can further improve upon these methods by selecting question subsets that will be maximally useful for prediction. Except in very data-poor settings, these approaches consistently achieve smaller prediction errors (in both MAE and RMSE), and greater ranking correlation between predicted and true scores (in both Spearman ρ and Kendall τ) across a range of benchmarks using both binary and continuous metrics. Furthermore, mRMR subsampling is much faster than competitor methods (which often involve fitting probabilistic models or running clustering algorithms), and is more likely to select the same questions under different random seeds or training data splits. Tutorial code can be found at https://github.com/sambowyer/mrmr_eval .
Related
- Transfer Learning with Heterogeneous Feature Spaces in Linear Regression
- Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression
- Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data
- Benign Overfitting Property of the Minimum ℓq -norm Interpolant Estimator in Regression Problem via Feature Space Decomposition
- Rare-Feature-Adaptive Subsampling for Big Data Regression
BibTeX
@misc{bowyer2026efficient,
title = {{Efficient Benchmarking Is Just Feature Selection and Multiple Regression}},
author = {Sam Bowyer and Acyr Locatelli and Kris Cao},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.25773},
eprint = {2605.25773},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.25773},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Efficient Benchmarking Is Just Feature Selection and Multiple Regression
AU - Sam Bowyer
AU - Acyr Locatelli
AU - Kris Cao
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2605.25773
UR - https://arxiv.org/abs/2605.25773
AB - Efficient benchmarking techniques aim to lower the computational cost of evaluating LLMs by predicting full benchmark scores using only a subset of a benchmark's questions. By reframing this problem as an instance of multiple regression with feature selection, we find that existing efficient benchmarking methods can be greatly improved by simply using kernel ridge regression at the prediction stage. Additionally, using an information-theoretic feature-selection algorithm called minimum redundancy maximum relevance (mRMR), we can further improve upon these methods by selecting question subsets that will be maximally useful for prediction. Except in very data-poor settings, these approaches consistently achieve smaller prediction errors (in both MAE and RMSE), and greater ranking correlation between predicted and true scores (in both Spearman ρ and Kendall τ) across a range of benchmarks using both binary and continuous metrics. Furthermore, mRMR subsampling is much faster than competitor methods (which often involve fitting probabilistic models or running clustering algorithms), and is more likely to select the same questions under different random seeds or training data splits. Tutorial code can be found at https://github.com/sambowyer/mrmr_eval .
ER -