A study of CMS analysis pipelines through the Integration Challenge
Abstract
The upcoming High-Luminosity Large Hadron Collider (HL-LHC) at CERN will deliver an unprecedented volume of data for High Energy Physics (HEP). This wealth of information offers significant opportunities for scientific discovery, but its scale challenges traditional analysis workflows. In this paper, we present CMS analysis pipelines being developed to meet HL-LHC demands. These pipelines build on the broader scientific Python ecosystem, complemented by solutions specifically designed for HEP. A central focus of this work is the Integration Challenge, an IRIS-HEP led effort aimed at assessing the readiness of developed software stack to be used in real world physics analysis and improving the readiness of analysis facilities for the HL-LHC era. The Integration Challenge acts as an end-to-end integration test: by implementing a complete physics analysis pipeline, it evaluates tool interoperability and the overall user experience for analysts. The current pipeline includes columnar data processing, machine learning, statistical inference, and visualization tasks covering a variety of CMS analysis scenarios. In addition, the Integration Challenge explores efficient strategies for delivering skimmed data using diverse tools and data formats, as well as evaluating the ServiceX data-delivery system for HEP analyses. Throughout the testing phase, we also investigated several prototype services, such as histogram-as-a-service capabilities, along with other emerging services that may support future HL-LHC analysis workflows.
Related
- Recent benchmarks in the Analysis Grand Challenge and integration with Combine (and HS3)
- Efficient biomass pyrolysis under sub-design solar input through SiC foam integration and process parameter optimization
- SGN: A python framework for stream-processing pipelines
- Towards Continuous Profiling and Optimization of Quantum-Classical Pipelines
- Open-source Software and Pipelines for Generalizable and Reproducible EEG Analysis: An Overview
BibTeX
@misc{aly2026study,
title = {{A study of CMS analysis pipelines through the Integration Challenge}},
author = {Mohamed Aly and Peter Elmer and Peter Fackeldey and Oksana Shadura},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.34698},
eprint = {2609.34698},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.34698},
}
RIS
TY - UNPB
TI - A study of CMS analysis pipelines through the Integration Challenge
AU - Mohamed Aly
AU - Peter Elmer
AU - Peter Fackeldey
AU - Oksana Shadura
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2609.34698
UR - https://arxiv.org/abs/2609.34698
AB - The upcoming High-Luminosity Large Hadron Collider (HL-LHC) at CERN will deliver an unprecedented volume of data for High Energy Physics (HEP). This wealth of information offers significant opportunities for scientific discovery, but its scale challenges traditional analysis workflows. In this paper, we present CMS analysis pipelines being developed to meet HL-LHC demands. These pipelines build on the broader scientific Python ecosystem, complemented by solutions specifically designed for HEP. A central focus of this work is the Integration Challenge, an IRIS-HEP led effort aimed at assessing the readiness of developed software stack to be used in real world physics analysis and improving the readiness of analysis facilities for the HL-LHC era. The Integration Challenge acts as an end-to-end integration test: by implementing a complete physics analysis pipeline, it evaluates tool interoperability and the overall user experience for analysts. The current pipeline includes columnar data processing, machine learning, statistical inference, and visualization tasks covering a variety of CMS analysis scenarios. In addition, the Integration Challenge explores efficient strategies for delivering skimmed data using diverse tools and data formats, as well as evaluating the ServiceX data-delivery system for HEP analyses. Throughout the testing phase, we also investigated several prototype services, such as histogram-as-a-service capabilities, along with other emerging services that may support future HL-LHC analysis workflows.
ER -