LLM Benchmarking via Representation Multi-task Learning
Abstract
Quantifying and evaluating the capabilities of Large Language Models (LLMs) remains a fundamental challenge in modern data science and artificial intelligence. In this paper, we consider LLM evaluation based on their performance across items in multiple benchmark domains (e.g., mathematical reasoning and coding) within a leaderboard framework. Our goal is to address two core questions: (1) How do we derive more accurate domain-specific scores by borrowing information across domains? and (2) How do we define and estimate an overall score that aggregates performance across multiple domains? To solve these problems, we propose a novel statistical framework based on representation Multi-task Learning (MTL) and an item response theory model. Specifically, we define overall and domain-specific LLM traits through an Item Response Theory (IRT) model, and propose an MTL approach to estimate these traits from item-level response data. We develop a computationally efficient estimator and establish its minimax optimality under certain asymptotic regimes. This framework provides a rigorous measurement foundation for systematic LLM evaluation. We conduct extensive simulations, demonstrating the superior performance of the proposed method over competing methods. Crucially for the Applications and Case Studies section, we apply the proposed framework to MMLU response data from the Hugging Face Open LLM Leaderboard, covering 4,272 LLMs and 13,232 items across 56 subjects. The empirical analysis reveals substantial heterogeneity in domain size and difficulty, together with strong positive cross-domain dependence, highlighting the practical value and substantive insights generated by our approach.
Related
- PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatorics
- Sharp Regret Bounds and a Task-Covariance Correction for Spectral Representation Learning
- A Multi‐Task Learning Approach for Identifying Prognostic Variables Across Multiple Historical Clinical Trials
- Robust Multi-Task Learning for Principal Component Analysis
- Scalable Multi-Task Inverse Reinforcement Learning
BibTeX
@misc{xie2026benchmarking,
title = {{LLM Benchmarking via Representation Multi-task Learning}},
author = {Yuqing Xie and Yuxuan Xu and Yang Feng and Yunxiao Chen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.05613},
eprint = {2610.05613},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.05613},
}
RIS
TY - UNPB
TI - LLM Benchmarking via Representation Multi-task Learning
AU - Yuqing Xie
AU - Yuxuan Xu
AU - Yang Feng
AU - Yunxiao Chen
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.05613
UR - https://arxiv.org/abs/2610.05613
AB - Quantifying and evaluating the capabilities of Large Language Models (LLMs) remains a fundamental challenge in modern data science and artificial intelligence. In this paper, we consider LLM evaluation based on their performance across items in multiple benchmark domains (e.g., mathematical reasoning and coding) within a leaderboard framework. Our goal is to address two core questions: (1) How do we derive more accurate domain-specific scores by borrowing information across domains? and (2) How do we define and estimate an overall score that aggregates performance across multiple domains? To solve these problems, we propose a novel statistical framework based on representation Multi-task Learning (MTL) and an item response theory model. Specifically, we define overall and domain-specific LLM traits through an Item Response Theory (IRT) model, and propose an MTL approach to estimate these traits from item-level response data. We develop a computationally efficient estimator and establish its minimax optimality under certain asymptotic regimes. This framework provides a rigorous measurement foundation for systematic LLM evaluation. We conduct extensive simulations, demonstrating the superior performance of the proposed method over competing methods. Crucially for the Applications and Case Studies section, we apply the proposed framework to MMLU response data from the Hugging Face Open LLM Leaderboard, covering 4,272 LLMs and 13,232 items across 56 subjects. The empirical analysis reveals substantial heterogeneity in domain size and difficulty, together with strong positive cross-domain dependence, highlighting the practical value and substantive insights generated by our approach.
ER -