Multitask Jet Analysis with Vision-Language Models: A Physics-Informed Four-Panel Representation
Abstract
The enormous recorded data at high-energy physics (HEP) colliders make the accurate identification of physics objects a bottleneck in disentangling event topologies, where machine learning has become a standard tool for jet tagging. In this work, we examine whether Vision-Language Models (VLMs) can provide a common interface for structured jet analysis from a single physics-informed image. Each JetClass jet becomes a 224x224 RGB image with four panels encoding p_T flow of all constituents; charged-hadron, neutral-hadron, and electromagnetic composition; p_T-weighted impact-parameter significances with displaced-track multiplicity; and signed-track p_T densities with local p_T^k-weighted jet-charge asymmetry. Four open VLMs are adapted with low-rank adaptation (LoRA) for class classification across QCD, Higgs, W/Z, and top jets, six-field attribute prediction, and cross-panel consistency with replaced-panel localization, evaluated on 24000 balanced JetClass test jets. Ablations identify the impact-parameter lifetime signature of heavy-flavor tagging as dominant, with jet charge separating the nearly mass-degenerate hadronic W and Z. Zero-shot recall stays near chance (3.33%-10.57%), once adapted, Task 1 Macro recall grows monotonically with training size, and Gemma4-E4B reaches 75.05% (74.87% F_1), 93.31% Task 2 field-mean, and 99.88%/99.75% binary/localization recall. Reallocating budget toward W/Z jets raises Zqq recall by up to 9.83 points at near-constant Macro recall. Transfer to broader-topology JetClass-II and real-data Aspen Open Jets (probing the simulation-to-data gap) is retained: with 3000 target jets, Task 1 recall exceeds 70% on JetClass-II and every Task 2/3 metric exceeds 92%. Thus rasterizing energy flow, species, displacement, and jet charge, with parameter-efficient adaptation, supports jet analysis through one instruction-conditioned interface for future collider data.
Related
- ViSSR: Physics-Informed Hybrid Vision-LSTM/State-Space Super-Resolution for Digital Rock Microstructure Analysis
- Accelerating noisy variational quantum algorithms with physics-informed denoising networks
- [SUPERSEDED] Physics-Informed AI and Quantum Technologies for Certifiable Real-Time Control of Compact Fusion Generators
- PINNIES: An efficient physics-informed neural network framework for integral operator problems
- Single-step initialization of a 16-channel silicon nitride DWDM demultiplexer using a physics-informed digital twin
BibTeX
@misc{zhang2026multitask,
title = {{Multitask Jet Analysis with Vision-Language Models: A Physics-Informed Four-Panel Representation}},
author = {Lu Zhang and Rachik Soualah and Abbes Amira},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.24334},
eprint = {2609.24334},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.24334},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Multitask Jet Analysis with Vision-Language Models: A Physics-Informed Four-Panel Representation
AU - Lu Zhang
AU - Rachik Soualah
AU - Abbes Amira
PY - 2026
DA - 2026/09/21
PB - arXiv
DO - 10.48550/arxiv.2609.24334
UR - https://arxiv.org/abs/2609.24334
AB - The enormous recorded data at high-energy physics (HEP) colliders make the accurate identification of physics objects a bottleneck in disentangling event topologies, where machine learning has become a standard tool for jet tagging. In this work, we examine whether Vision-Language Models (VLMs) can provide a common interface for structured jet analysis from a single physics-informed image. Each JetClass jet becomes a 224x224 RGB image with four panels encoding p_T flow of all constituents; charged-hadron, neutral-hadron, and electromagnetic composition; p_T-weighted impact-parameter significances with displaced-track multiplicity; and signed-track p_T densities with local p_T^k-weighted jet-charge asymmetry. Four open VLMs are adapted with low-rank adaptation (LoRA) for class classification across QCD, Higgs, W/Z, and top jets, six-field attribute prediction, and cross-panel consistency with replaced-panel localization, evaluated on 24000 balanced JetClass test jets. Ablations identify the impact-parameter lifetime signature of heavy-flavor tagging as dominant, with jet charge separating the nearly mass-degenerate hadronic W and Z. Zero-shot recall stays near chance (3.33%-10.57%), once adapted, Task 1 Macro recall grows monotonically with training size, and Gemma4-E4B reaches 75.05% (74.87% F_1), 93.31% Task 2 field-mean, and 99.88%/99.75% binary/localization recall. Reallocating budget toward W/Z jets raises Zqq recall by up to 9.83 points at near-constant Macro recall. Transfer to broader-topology JetClass-II and real-data Aspen Open Jets (probing the simulation-to-data gap) is retained: with 3000 target jets, Task 1 recall exceeds 70% on JetClass-II and every Task 2/3 metric exceeds 92%. Thus rasterizing energy flow, species, displacement, and jet charge, with parameter-efficient adaptation, supports jet analysis through one instruction-conditioned interface for future collider data.
ER -