CHARIZARD: A High-throughput Generic Pipeline for Present and Upcoming Interferometers
Abstract
Radio interferometers now deliver wide bandwidths and high sensitivity, and with them data volumes that are awkward to reduce by hand and slow to reduce automatically. The limit is not always the algorithms themselves but the way the work is laid out across the computing available. We present CHARIZARD, a pipeline that carries a raw measurement set through flagging, cross-calibration, self-calibration and imaging to full-Stokes images, and which is built around distributing that work. The observing band is repartitioned into sub-bands that stay independent through cross-calibration, the unit of work becomes the target field and sub-band pair at self-calibration, and stages touching disjoint data are run together. Each stage is an ordinary PBS or SLURM job, submitted and tracked by a scheduler-agnostic layer, and every computation runs inside a single rootless container, so a user holding no administrative rights gets the same behaviour on any cluster. We benchmark the pipeline on MeerKAT measurements with full polarization calibration. Residual on-axis instrumental polarization is $0.13$ per cent in linear and $0.06$ per cent in circular; the deepest field reaches 4.11 μJy~beam−1 over $787$~MHz with σI/σV = 1.010; positions agree with RACS-low to better than 0.4arcsec and flux densities to a median ratio of $1.06$. A $1.5$~TB track completes $48.6$ core-hours of work in $18.4$~hours with self-calibration and polarization calibration. The next generation of the radio telescopes will not be short of data. CHARIZARD attempts to show how the distribution of the work provides a useful alternative for handling the staggering processing requirements.
Related
- Quasars Microlensed by Milky Way Stars: Two Upcoming Events Predicted with Gaia Data Release 3
- Cosmology with the angular cross-correlation of gravitational-wave and galaxy catalogs: Forecasts for next-generation interferometers and the Euclid survey
- Direct multi-model dark-matter search with gravitational-wave interferometers using data from the first part of the fourth LIGO-Virgo-KAGRA observing run
- CtaAgnVar : a Pipeline for Variability Studies with the Cherenkov Telescope Array Observatory
- meer21cm: an Analysis Pipeline and Comprehensive Toolkit for HI Intensity Mapping
BibTeX
@misc{pal2026charizard,
title = {{CHARIZARD: A High-throughput Generic Pipeline for Present and Upcoming Interferometers}},
author = {Arpan Pal and Ruta Kale},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.21158},
eprint = {2610.21158},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.21158},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - CHARIZARD: A High-throughput Generic Pipeline for Present and Upcoming Interferometers
AU - Arpan Pal
AU - Ruta Kale
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.21158
UR - https://arxiv.org/abs/2610.21158
AB - Radio interferometers now deliver wide bandwidths and high sensitivity, and with them data volumes that are awkward to reduce by hand and slow to reduce automatically. The limit is not always the algorithms themselves but the way the work is laid out across the computing available. We present CHARIZARD, a pipeline that carries a raw measurement set through flagging, cross-calibration, self-calibration and imaging to full-Stokes images, and which is built around distributing that work. The observing band is repartitioned into sub-bands that stay independent through cross-calibration, the unit of work becomes the target field and sub-band pair at self-calibration, and stages touching disjoint data are run together. Each stage is an ordinary PBS or SLURM job, submitted and tracked by a scheduler-agnostic layer, and every computation runs inside a single rootless container, so a user holding no administrative rights gets the same behaviour on any cluster. We benchmark the pipeline on MeerKAT measurements with full polarization calibration. Residual on-axis instrumental polarization is $0.13$ per cent in linear and $0.06$ per cent in circular; the deepest field reaches 4.11 μJy~beam^−1 over $787$~MHz with σ_I/σ_V = 1.010; positions agree with RACS-low to better than 0.4arcsec and flux densities to a median ratio of $1.06$. A $1.5$~TB track completes $48.6$ core-hours of work in $18.4$~hours with self-calibration and polarization calibration. The next generation of the radio telescopes will not be short of data. CHARIZARD attempts to show how the distribution of the work provides a useful alternative for handling the staggering processing requirements.
ER -