The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean
Abstract
Agent benchmarks are increasingly used to compare large language models (LLMs) and guide deployment decisions, yet benchmark scores are meaningful only if they measure model capability rather than properties of the evaluation pipeline. We identify a double measurement confound: execution-critical decisions are performed by a fixed scaffold instead of the model, while the scorer evaluates outputs using criteria that may not reflect task correctness. We unify these issues within a measurement-theoretic framework that characterizes when benchmark scores can be interpreted as evidence of model capability, and instantiate it with an audit-and-repair protocol that (i) transfers execution-critical decisions from the scaffold to the model, (ii) replaces shape-based evaluation with seeded ground-truth scoring, and (iii) reports reliability beyond the mean through worst-case and tail-risk metrics. Experiments on ComtradeBench show that the joint intervention transforms a nearly flat leaderboard into a reliability spectrum that distinguishes both average performance and robustness across seeds. Applying the audit to existing benchmarks further shows that scorer validity is benchmark-specific, whereas scaffold ownership is an uncontrolled axis wherever we probed it. Our results suggest that benchmark scores should be interpreted together with their scaffolding level, scoring criterion, and reliability profile, providing a practical framework for more valid evaluation of LLM agents.
Related
- Do Quantum Measurement Advantages Survive to Astrophysical Inference? Seven Benchmarks in Optical Interferometry and Imaging
- Impact of dephased entanglement, mixed input states and varying measurement orientations on the reliability of cryptographic keys generated via the quantum protocol E91
- Spatial Colorimetric Mapping of pH and Dissolved Carbon during Convective Dissolution of CO{\_2}: Constraints from Domain, Geometry, and Indicator Mix on Measurement Reliability
- A proposed E 8 × E 8 kinematic scaffolding for the standard model with pre-gravitation
- The Whittaker-Maxwell Scaffolding. Scaffolding or template: what remains of a model when it is removed
BibTeX
@misc{zhang2026double,
title = {{The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean}},
author = {Yonghong Zhang and Shadi Motaali and Vu Phong Dinh and Avin Piroutiniya and Jorge E. López de Vergara and Luis de Pedro and Ricardo Correia and Isabel M. Parra and Yong Xie},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.09218},
eprint = {2609.09218},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.09218},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean
AU - Yonghong Zhang
AU - Shadi Motaali
AU - Vu Phong Dinh
AU - Avin Piroutiniya
AU - Jorge E. López de Vergara
AU - Luis de Pedro
AU - Ricardo Correia
AU - Isabel M. Parra
AU - Yong Xie
PY - 2026
DA - 2026/09/06
PB - arXiv
DO - 10.48550/arxiv.2609.09218
UR - https://arxiv.org/abs/2609.09218
AB - Agent benchmarks are increasingly used to compare large language models (LLMs) and guide deployment decisions, yet benchmark scores are meaningful only if they measure model capability rather than properties of the evaluation pipeline. We identify a double measurement confound: execution-critical decisions are performed by a fixed scaffold instead of the model, while the scorer evaluates outputs using criteria that may not reflect task correctness. We unify these issues within a measurement-theoretic framework that characterizes when benchmark scores can be interpreted as evidence of model capability, and instantiate it with an audit-and-repair protocol that (i) transfers execution-critical decisions from the scaffold to the model, (ii) replaces shape-based evaluation with seeded ground-truth scoring, and (iii) reports reliability beyond the mean through worst-case and tail-risk metrics. Experiments on ComtradeBench show that the joint intervention transforms a nearly flat leaderboard into a reliability spectrum that distinguishes both average performance and robustness across seeds. Applying the audit to existing benchmarks further shows that scorer validity is benchmark-specific, whereas scaffold ownership is an uncontrolled axis wherever we probed it. Our results suggest that benchmark scores should be interpreted together with their scaffolding level, scoring criterion, and reliability profile, providing a practical framework for more valid evaluation of LLM agents.
ER -