How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?
Abstract
Jet tagging is one of the most important applications of machine learning at the Large Hadron Collider. Recent models build in more physics knowledge or rely on large-scale pretraining, but the experiments place strict limits on their inference cost. Knowledge distillation can transfer their performance to small students that are cheap to run. We ask which properties of a teacher help the student, how small a student can get and how close it can come to the state of the art. On a jet tagging benchmark, we distil six teachers that differ in architecture, symmetry and pretraining, including two foundation models, into five small students. Every teacher improves every student, and the gain depends mainly on the strength of the teacher. We then shrink our two strongest distilled students, Transformer and L-GATr-slim, in width and in depth. With 20 thousand parameters, a distilled L-GATr-slim outperforms every published jet tagger trained from scratch at a fraction of the inference cost of every teacher. Even with two thousand parameters, it matches ParT, a model with a thousand times as many parameters. We also study what the students learn from their teachers. Our benchmark of inference cost shows that students of the same size can differ by almost an order of magnitude in speed. We release the code and the weights of all trained models.
Related
- Foundation-Model Supervision of Distilled Potentials for Million-Atom Molecular Dynamics
- This is SParTA: A modular, efficient and scalable transformer architecture for jet-tagging
- Physics Foundation Models for Efficient Control of Fluid Flows
- Synthesis and characterization of size-controllable titanium dioxide nanocolloids via pulsed laser ablation in distilled water
- Follow-up comments to searches for QCD instantons via forward proton tagging
BibTeX
@misc{agarwal2026foundation,
title = {{How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?}},
author = {Jay Agarwal and Siddharth Khare and Dhruv Kumar},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.29439},
eprint = {2610.29439},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.29439},
}
RIS
TY - UNPB
TI - How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?
AU - Jay Agarwal
AU - Siddharth Khare
AU - Dhruv Kumar
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.29439
UR - https://arxiv.org/abs/2610.29439
AB - Jet tagging is one of the most important applications of machine learning at the Large Hadron Collider. Recent models build in more physics knowledge or rely on large-scale pretraining, but the experiments place strict limits on their inference cost. Knowledge distillation can transfer their performance to small students that are cheap to run. We ask which properties of a teacher help the student, how small a student can get and how close it can come to the state of the art. On a jet tagging benchmark, we distil six teachers that differ in architecture, symmetry and pretraining, including two foundation models, into five small students. Every teacher improves every student, and the gain depends mainly on the strength of the teacher. We then shrink our two strongest distilled students, Transformer and L-GATr-slim, in width and in depth. With 20 thousand parameters, a distilled L-GATr-slim outperforms every published jet tagger trained from scratch at a fraction of the inference cost of every teacher. Even with two thousand parameters, it matches ParT, a model with a thousand times as many parameters. We also study what the students learn from their teachers. Our benchmark of inference cost shows that students of the same size can differ by almost an order of magnitude in speed. We release the code and the weights of all trained models.
ER -