HEPTABench: A framework for portable and controlled agentic evaluations in high energy physics
Abstract
Quantifying the capability and reliability of agentic systems in high-energy physics (HEP) requires repeated evaluations under controlled configurations. Establishing whether changes to models, tools, or harnesses improve performance also requires these components to be varied independently. We present the High Energy Physics Tool-and-Agent Benchmarking (HEPTABench) framework for portable and controlled evaluations of agentic systems, with separately configurable tasks, scaffolding, models, providers, harnesses, and domain capabilities. The framework combines HEP benchmark tasks and scientific rubrics with experiment execution and versioned tool interfaces. To support inspection, reproducibility, and post-hoc grading, the framework prioritizes recording and reporting complete experiment specifications, agent trajectories, and experiment artifacts. We demonstrate the framework in two benchmark campaigns. The first campaign evaluates a forward long-lived-particle reach calculation across four frontier coding-agent systems, comparing their built-in coding tools with the same tools supplemented by HEP domain tools. We show that including these HEP domain tools improves numerical accuracy and reduces trial-to-trial dispersion for every tested system. In a complementary campaign covering eight symbolic two-body decay tasks, we show that adding symbolic domain tools increases the fraction of successful trials for Claude Haiku 4.5 from 8% to 79%, while GPT-5.5 succeeds in every trial with built-in tools alone. Through systematic evaluation of agentic systems, HEPTABench enables the identification of factors that improve the accuracy and reliability of their scientific work.
Related
- ORB5X: a performance-portable global electromagnetic gyrokinetic PIC code in C++/Kokkos built using Agentic AI
- OPALX: A performance-portable framework for particle accelerator simulations
- Aitomia: An Agentic Framework for AI-Driven Atomistic and Quantum Chemical Simulations
- Automated Many-Body Simulations of Strongly Correlated Systems Using a Correlation-Aware Agentic Framework
- An Extensible Agentic Framework for Expert-Level Automation of Atomistic Simulations
BibTeX
@misc{menzo2026heptabench,
title = {{HEPTABench: A framework for portable and controlled agentic evaluations in high energy physics}},
author = {Tony Menzo and Alexander Roman and George T. Fleming and Sergei Gleyzer and Konstantin Matchev and Stephen Mrenna},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.18454},
eprint = {2610.18454},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.18454},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - HEPTABench: A framework for portable and controlled agentic evaluations in high energy physics
AU - Tony Menzo
AU - Alexander Roman
AU - George T. Fleming
AU - Sergei Gleyzer
AU - Konstantin Matchev
AU - Stephen Mrenna
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2610.18454
UR - https://arxiv.org/abs/2610.18454
AB - Quantifying the capability and reliability of agentic systems in high-energy physics (HEP) requires repeated evaluations under controlled configurations. Establishing whether changes to models, tools, or harnesses improve performance also requires these components to be varied independently. We present the High Energy Physics Tool-and-Agent Benchmarking (HEPTABench) framework for portable and controlled evaluations of agentic systems, with separately configurable tasks, scaffolding, models, providers, harnesses, and domain capabilities. The framework combines HEP benchmark tasks and scientific rubrics with experiment execution and versioned tool interfaces. To support inspection, reproducibility, and post-hoc grading, the framework prioritizes recording and reporting complete experiment specifications, agent trajectories, and experiment artifacts. We demonstrate the framework in two benchmark campaigns. The first campaign evaluates a forward long-lived-particle reach calculation across four frontier coding-agent systems, comparing their built-in coding tools with the same tools supplemented by HEP domain tools. We show that including these HEP domain tools improves numerical accuracy and reduces trial-to-trial dispersion for every tested system. In a complementary campaign covering eight symbolic two-body decay tasks, we show that adding symbolic domain tools increases the fraction of successful trials for Claude Haiku 4.5 from 8% to 79%, while GPT-5.5 succeeds in every trial with built-in tools alone. Through systematic evaluation of agentic systems, HEPTABench enables the identification of factors that improve the accuracy and reliability of their scientific work.
ER -