HEPToolBench 1.2: Testing How Reliably Language Models Can Drive Particle Physics Software
Abstract
Scientists increasingly want to drive research software by natural-language request, but fluent output helps only if it becomes a correct machine-readable artifact. We introduce HEPToolBench, a benchmark of 28 collider-simulation tasks scored by deterministic, task-specific scorers, plus a three-task structured-debugging extension. We evaluate 42 deployments, from small locally served open-weight models to hosted frontier systems. The central experiment compares direct generation of native HEP-tool syntax with a schema-mediated interface where the model returns a typed representation that deterministic software serializes. Across five matched requests, the mean score rises from 0.418 to 0.902 and task passes from 21/210 to 159/210, with 41 of 42 deployments improving; eleven deployments, seven locally served open-weight models, go from 0/5 passes under native syntax to 5/5 under the structured interface. Values use the corrected v1.2.1 contract, in which the two native scorers that had enforced operational conventions absent from their own prompts are rescored prompt-faithfully cohort-wide. One asymmetry remains: native scorers score the recorded response verbatim and reject Markdown-fenced replies, whereas structured scorers recover JSON from such wrappers; no full-cohort rescoring under a single symmetric extraction rule has been done. An audit of archived responses from 17 locally served deployments shows this does not explain their gains: passes remain six to ten times more frequent under the structured interface when both sides use the same extraction rule. A task pass alone does not guarantee runtime or scientific viability. Within this scope, moving syntax generation into deterministic software can substantially improve reliability for both small local and frontier models. Prompts, scorers, responses, and regeneration scripts are released for independent evaluation and extension.
Related
- An extensible software platform for automated operational and characterization testing of scientific cameras
- Noise-Awareness in Quantum Software Testing: An Empirical Landscape Analysis
- Efficiently and Reliably Measuring Information Processing Capacity in Dynamical Systems via Kernels
- Reduced latent leakage does not reliably predict lower likelihood bias in collider inference
- Magpie: A Graphical Magnon Software
BibTeX
@misc{singh2026heptoolbench,
title = {{HEPToolBench 1.2: Testing How Reliably Language Models Can Drive Particle Physics Software}},
author = {Aadarsh Singh and Sudhir K. Vempati},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.28232},
eprint = {2608.28232},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.28232},
}
RIS
TY - UNPB
TI - HEPToolBench 1.2: Testing How Reliably Language Models Can Drive Particle Physics Software
AU - Aadarsh Singh
AU - Sudhir K. Vempati
PY - 2026
DA - 2026/08/28
PB - arXiv
DO - 10.48550/arxiv.2608.28232
UR - https://arxiv.org/abs/2608.28232
AB - Scientists increasingly want to drive research software by natural-language request, but fluent output helps only if it becomes a correct machine-readable artifact. We introduce HEPToolBench, a benchmark of 28 collider-simulation tasks scored by deterministic, task-specific scorers, plus a three-task structured-debugging extension. We evaluate 42 deployments, from small locally served open-weight models to hosted frontier systems. The central experiment compares direct generation of native HEP-tool syntax with a schema-mediated interface where the model returns a typed representation that deterministic software serializes. Across five matched requests, the mean score rises from 0.418 to 0.902 and task passes from 21/210 to 159/210, with 41 of 42 deployments improving; eleven deployments, seven locally served open-weight models, go from 0/5 passes under native syntax to 5/5 under the structured interface. Values use the corrected v1.2.1 contract, in which the two native scorers that had enforced operational conventions absent from their own prompts are rescored prompt-faithfully cohort-wide. One asymmetry remains: native scorers score the recorded response verbatim and reject Markdown-fenced replies, whereas structured scorers recover JSON from such wrappers; no full-cohort rescoring under a single symmetric extraction rule has been done. An audit of archived responses from 17 locally served deployments shows this does not explain their gains: passes remain six to ten times more frequent under the structured interface when both sides use the same extraction rule. A task pass alone does not guarantee runtime or scientific viability. Within this scope, moving syntax generation into deterministic software can substantially improve reliability for both small local and frontier models. Prompts, scorers, responses, and regeneration scripts are released for independent evaluation and extension.
ER -