Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks
Abstract
Reconstruction and simulation at a collider experiment are long chains of specialised algorithms, each tuned to a single step. We explore how one model can serve many of those steps at once, while still producing the intermediate objects (tracks, calorimeter cells, clusters, particles and jets) that make the chain interpretable. To do so, we represent every object in a jet in a single shared token vocabulary and train one model to map any subset of these modalities to any other. A task is then only a choice of which modalities to provide and which to request: particle flow, detector simulation and charged energy subtraction are all directions through the same set of weights. We train over all modality combinations, with a decoder emitting tokens either autoregressively or in parallel. With tokenisation, both architectures train stably with little tuning. In evaluations, both models produce realistic reconstruction and simulation objects, with the autoregressive model particularly faithful to output from Geant4. The autoregressive model also outperforms a state-of-the-art particle flow algorithm on many typical jet reconstruction metrics.
Related
- Track finding and particle identification performance with particle physics foundation models
- Leveraging Industrial Foundation Models at the Edge of Particle Physics Detectors via Distillation Learning and Hardware Co-design
- Panda Diplomacy: Foundation Model Pre-training across Particle Imaging Detectors for High Energy and Nuclear Physics
- Towards Foundation Models on Hardware Accelerators for Particle Physics
- Machine-learned particle flow as a foundation model for collider physics
BibTeX
@misc{kakati2026prompting,
title = {{Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks}},
author = {Nilotpal Kakati and Daniel Murnane and Baran Hashemi and Samuel Klein and Jeffrey Krupa and Eilam Gross and Lukas Heinrich and Michael Kagan},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.31862},
eprint = {2609.31862},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.31862},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks
AU - Nilotpal Kakati
AU - Daniel Murnane
AU - Baran Hashemi
AU - Samuel Klein
AU - Jeffrey Krupa
AU - Eilam Gross
AU - Lukas Heinrich
AU - Michael Kagan
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.31862
UR - https://arxiv.org/abs/2609.31862
AB - Reconstruction and simulation at a collider experiment are long chains of specialised algorithms, each tuned to a single step. We explore how one model can serve many of those steps at once, while still producing the intermediate objects (tracks, calorimeter cells, clusters, particles and jets) that make the chain interpretable. To do so, we represent every object in a jet in a single shared token vocabulary and train one model to map any subset of these modalities to any other. A task is then only a choice of which modalities to provide and which to request: particle flow, detector simulation and charged energy subtraction are all directions through the same set of weights. We train over all modality combinations, with a decoder emitting tokens either autoregressively or in parallel. With tokenisation, both architectures train stably with little tuning. In evaluations, both models produce realistic reconstruction and simulation objects, with the autoregressive model particularly faithful to output from Geant4. The autoregressive model also outperforms a state-of-the-art particle flow algorithm on many typical jet reconstruction metrics.
ER -