Beyond a Better Score: Long-Horizon Agentic ML Development and Evaluation Protocol for Physics Time Series

Abstract

Physics time-series data are central to fundamental discoveries involving dark matter, neutrinos, and gravitational waves. Machine learning has shown strong potential to recover weak scientific signals buried in complex noise. Recent LLM-agent systems further automate ML development, but existing approaches are often driven primarily by one or a few numerical scores. In pursuit of a better score, agents may exploit shortcuts, trigger mode collapse, or game the evaluation procedure, producing models that achieve high scores but are scientifically invalid. We introduce SIDERIUS, an LLM agent system designed to move beyond a better score toward scientifically valid, long-horizon ML development. SIDERIUS combines contract-defined composable capabilities for flexible orchestration and human-in-the-loop, a multi-layer scientific evaluation protocol that distinguishes valid models from unhealthy or gaming models, and resource-aware exploration for computationally expensive scientific ML. In our TIDMAD benchmark, SIDERIUS with a GPT-based orchestrator achieves the highest Valid-model proposal rate of 63.8% and the best valid denoising score of 2.744, compared with 1.6% and -0.647, respectively, for the same GPT model using the general-purpose Codex coding agent. We further benchmark SIDERIUS on three additional physics time-series datasets, demonstrating its reuse and performance across different tasks. Together, these results suggest that progress in agentic scientific ML should be measured beyond a better score, by how an agent explores and by whether the models it produces can be trusted as science.

Reining in an Agentic Harness for High Energy Physics
Tony Menzo, George T. Fleming, Konstantin T. Matchev +2
arXiv1 Sep 2026PreprintCC BY 4.0Full textHigh-energy & particle physicsarXiv
SolutionsX: supergravity theories/solutions as code, with an agentic workflow for machine-verifiable physics
Vasil Dimitrov
arXiv7 Sep 2026PreprintCC BY 4.0Full textGravitation & general relativity High-energy & particle physicsarXiv
HEPTABench: A framework for portable and controlled agentic evaluations in high energy physics
Tony Menzo, Alexander Roman, George T. Fleming +3
arXiv13 Oct 2026PreprintCC BY 4.0Full textHigh-energy & particle physicsarXiv
Solver Agent: an Agentic AI Framework for Theoretical Physics Computations Applied to F-theory Uplifts of O3-planes and S-folds
Eliott Morgensztern, Cesar Fierro Cota, Alessandro Mininno
HAL28 Sep 2026PreprintarXiv licenceFull textAlgebra & number theory Mathematical physicsHALarXiv
Principles and Techniques for Agentic Research in Experimental Physics
Niko R. Reed, Declan M. Daly, Andrew J. Beling +4
arXiv12 Oct 2026PreprintCC BY 4.0Full textComputational physics Quantum physics & quantum informationarXiv

Cite

BibTeX RIS

BibTeX

@misc{ma2026beyond,
  title = {{Beyond a Better Score: Long-Horizon Agentic ML Development and Evaluation Protocol for Physics Time Series}},
  author = {Yue Ma and Lucas Venetoulias and Akbota Assan and Aobo Li},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.14024},
  eprint = {2610.14024},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.14024},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Beyond a Better Score: Long-Horizon Agentic ML Development and Evaluation Protocol for Physics Time Series
AU  - Yue Ma
AU  - Lucas Venetoulias
AU  - Akbota Assan
AU  - Aobo Li
PY  - 2026
DA  - 2026/10/01
PB  - arXiv
DO  - 10.48550/arxiv.2610.14024
UR  - https://arxiv.org/abs/2610.14024
AB  - Physics time-series data are central to fundamental discoveries involving dark matter, neutrinos, and gravitational waves. Machine learning has shown strong potential to recover weak scientific signals buried in complex noise. Recent LLM-agent systems further automate ML development, but existing approaches are often driven primarily by one or a few numerical scores. In pursuit of a better score, agents may exploit shortcuts, trigger mode collapse, or game the evaluation procedure, producing models that achieve high scores but are scientifically invalid. We introduce SIDERIUS, an LLM agent system designed to move beyond a better score toward scientifically valid, long-horizon ML development. SIDERIUS combines contract-defined composable capabilities for flexible orchestration and human-in-the-loop, a multi-layer scientific evaluation protocol that distinguishes valid models from unhealthy or gaming models, and resource-aware exploration for computationally expensive scientific ML. In our TIDMAD benchmark, SIDERIUS with a GPT-based orchestrator achieves the highest Valid-model proposal rate of 63.8% and the best valid denoising score of 2.744, compared with 1.6% and -0.647, respectively, for the same GPT model using the general-purpose Codex coding agent. We further benchmark SIDERIUS on three additional physics time-series datasets, demonstrating its reuse and performance across different tasks. Together, these results suggest that progress in agentic scientific ML should be measured beyond a better score, by how an agent explores and by whether the models it produces can be trusted as science.
ER  -

Abstract

Related

Cite