How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
Abstract
Much of the recent progress in machine learning domains such as language models has come from scaling laws that predict performance as a function of training effort. In high-energy physics (HEP) similar behavior has now been observed. To aid further study, we present a systematic procedure to derive robust scaling laws and compare design choices on the relevant budget axes for HEP tasks. We first validate the full scaling trajectory on toy problems and then apply the procedure to multi-task transformers on the ~11 billion-jet ATLAS JetSet2 dataset, in both the compute- and data-constrained regimes. For the latter, we predict, to the best of our knowledge for the first time, the jointly optimal model size, training horizon, learning rate and batch size under early stopping. At compute-optimal scaling, we recover a near-equal √C dependence of model and dataset size, and find that auxiliary objectives lower the primary jet-classification loss at equal compute budget. Expanding the inputs toward lower-level data systematically lowers the loss while leaving the scaling exponent nearly unchanged. The onset of the power-law regime is itself set by scale: below a threshold in dataset size the loss carries little information about high-compute scaling, underscoring the value of large, high-quality full-simulation datasets as a foundation for scaling studies and the development of foundation models in HEP.
Related
- Big Crunching Your Big Bang: Doubly Warped Time-Dependent Type II Flux Backgrounds and Scale Separation
- A robust digital receiver architecture for continuous-variable quantum key distribution
- Quantum computers will not be that different: A blueprint for quantum computer architecture at scale
- From phase analysis to phase tracking: a robust approach to atomic-scale strain mapping
- SCALE: Simulation-Calibrated Amortized Learning for Energy Materials (A hybrid architecture connecting deterministic modeling, real-world data, and transformer-scale inference for accelerated energy-materials discovery)
BibTeX
@misc{vigl2026scale,
title = {{How to scale your HEP ML models: A recipe for robust architecture comparisons at scale}},
author = {Matthias Vigl and Nikita Pond and Jackson Barr and Alexander Froch and Dan Guest and Nicole Hartman and Michael Kagan and Lukas Heinrich},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06784},
eprint = {2610.06784},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.06784},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
AU - Matthias Vigl
AU - Nikita Pond
AU - Jackson Barr
AU - Alexander Froch
AU - Dan Guest
AU - Nicole Hartman
AU - Michael Kagan
AU - Lukas Heinrich
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.06784
UR - https://arxiv.org/abs/2610.06784
AB - Much of the recent progress in machine learning domains such as language models has come from scaling laws that predict performance as a function of training effort. In high-energy physics (HEP) similar behavior has now been observed. To aid further study, we present a systematic procedure to derive robust scaling laws and compare design choices on the relevant budget axes for HEP tasks. We first validate the full scaling trajectory on toy problems and then apply the procedure to multi-task transformers on the ~11 billion-jet ATLAS JetSet2 dataset, in both the compute- and data-constrained regimes. For the latter, we predict, to the best of our knowledge for the first time, the jointly optimal model size, training horizon, learning rate and batch size under early stopping. At compute-optimal scaling, we recover a near-equal √C dependence of model and dataset size, and find that auxiliary objectives lower the primary jet-classification loss at equal compute budget. Expanding the inputs toward lower-level data systematically lowers the loss while leaving the scaling exponent nearly unchanged. The onset of the power-law regime is itself set by scale: below a threshold in dataset size the loss carries little information about high-compute scaling, underscoring the value of large, high-quality full-simulation datasets as a foundation for scaling studies and the development of foundation models in HEP.
ER -