Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
Abstract
Mixture-of-Experts (MoE) models can increase parameter capacity without proportionally increasing active computation, but it is unclear how this trade-off behaves in particle-physics transformers. We study dense and MoE Particle Transformers on 188-class JetClass-II, varying expert count, routing capacity, top-K, and auxiliary loss. We find that, when token dropping is avoided, top-1 MoE models improve over the dense baseline at nearly unchanged nominal forward compute, while further increasing the number of stored experts produces little additional accuracy gain. Activating multiple experts per token yields additional predictive improvements at higher computational cost. Routing analyses show that expert assignments become more strongly associated with particle identity and kinematics in some configurations, but this structure does not increase monotonically with classification performance. These results highlight the need to distinguish stored parameter capacity, active computation, routing capacity, and routing organization when evaluating sparse expert models for jet classification. Code and experiment configurations are available at https://github.com/kpendiyala/MPT.
Related
- End-to-end Particle Reconstruction with Transformers in High-Granularity Calorimeters
- Particle transformers for identifying Lorentz-boosted Higgs bosons decaying to a pair of W bosons
- A unified gas-kinetic wave-particle method for multiscale gas-mixture flow with an elementary chemical reaction
- Interacting Tomonaga-Lüttinger liquid with impurity for interaction constant K = 1/2: Thermopower investigation, entropy variation and heat capacity densities associated with thermoelectric particle transport
- Measure-to-Measure Interpolation Using Transformers
BibTeX
@misc{pendiyala2026conditional,
title = {{Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers}},
author = {Kaushik Pendiyala and Haris Zia and Trevin Lee and Timothy Legge and Alejandro J. De Leon and Zihan Zhao and Aaron Wang and Abhijith Gandrakota and Jennifer Ngadiuba and Richard Cavanaugh and Javier Duarte},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02701},
eprint = {2610.02701},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02701},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
AU - Kaushik Pendiyala
AU - Haris Zia
AU - Trevin Lee
AU - Timothy Legge
AU - Alejandro J. De Leon
AU - Zihan Zhao
AU - Aaron Wang
AU - Abhijith Gandrakota
AU - Jennifer Ngadiuba
AU - Richard Cavanaugh
AU - Javier Duarte
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.02701
UR - https://arxiv.org/abs/2610.02701
AB - Mixture-of-Experts (MoE) models can increase parameter capacity without proportionally increasing active computation, but it is unclear how this trade-off behaves in particle-physics transformers. We study dense and MoE Particle Transformers on 188-class JetClass-II, varying expert count, routing capacity, top-K, and auxiliary loss. We find that, when token dropping is avoided, top-1 MoE models improve over the dense baseline at nearly unchanged nominal forward compute, while further increasing the number of stored experts produces little additional accuracy gain. Activating multiple experts per token yields additional predictive improvements at higher computational cost. Routing analyses show that expert assignments become more strongly associated with particle identity and kinematics in some configurations, but this structure does not increase monotonically with classification performance. These results highlight the need to distinguish stored parameter capacity, active computation, routing capacity, and routing organization when evaluating sparse expert models for jet classification. Code and experiment configurations are available at https://github.com/kpendiyala/MPT.
ER -