Efficient searches of small signals across two-channel noisy data: a challenge in Big Data observations
Abstract
Searching for novel small signals in noisy data is preferably pursued by correlation in two or more independently operating channels. Signals of potential interest exist in tails beyond κσ, where κ denotes a multiple of the standard deviation σ of the data. Since moving data is a major cost factor in Big Data analysis and heterogeneous computing more generally, efficiency may be optimized by restricting the correlations computation to the tails of two-channel data exceeding κσ. Already, a moderate value κ≳2 realizes a data-reduction by at least an order of magnitude. Here, we study this approach using a novel {\it Excess Probability Ratio} (EPR), correlating Boolean data resulting from tails beyond a cut-off κσ. We compare and rank EPR performance against conventional direct cross-correlation (DCC) and Pearson coefficient (PC), applicable to the original data with no cut-off. This benchmark is performed over different combinations of background noise Gaussian, Poisson and Uniform and signals Gaussian, Poisson, Uniform, Chirps and Sine waves. Results show performance of EPR to be comparable to that of PC, providing a new approach for significant improvements in efficiency with essentially no loss of sensitivity, relevant to the present era of Big Data observatories.
Versions
Related
- Recycling serendipitous observations of XRISM for dark matter searches
- Red noise and evolving signals: a complete frequentist approach to supermassive black hole binary searches with pulsar timing arrays
- NEXUS: Transient Searches and First Results from Year One Observations
- Cosmic Ray Observations using Coincident Scintillator and Radio Signals with the Beamforming Elevated Array for COsmic Neutrinos (BEACON)
- The pulsar population of Omega Centauri: New TRAPUM searches, timing, and cluster dynamics with MeerKAT and Parkes
BibTeX
@misc{abchouyeh2026efficient,
title = {{Efficient searches of small signals across two-channel noisy data: a challenge in Big Data observations}},
author = {Maryam Aghaei Abchouyeh and Maurice H. P. M. van Putten},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.00013},
eprint = {2609.00013},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.00013},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Efficient searches of small signals across two-channel noisy data: a challenge in Big Data observations
AU - Maryam Aghaei Abchouyeh
AU - Maurice H. P. M. van Putten
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.00013
UR - https://arxiv.org/abs/2609.00013
AB - Searching for novel small signals in noisy data is preferably pursued by correlation in two or more independently operating channels. Signals of potential interest exist in tails beyond κσ, where κ denotes a multiple of the standard deviation σ of the data. Since moving data is a major cost factor in Big Data analysis and heterogeneous computing more generally, efficiency may be optimized by restricting the correlations computation to the tails of two-channel data exceeding κσ. Already, a moderate value κ≳2 realizes a data-reduction by at least an order of magnitude. Here, we study this approach using a novel {\it Excess Probability Ratio} (EPR), correlating Boolean data resulting from tails beyond a cut-off κσ. We compare and rank EPR performance against conventional direct cross-correlation (DCC) and Pearson coefficient (PC), applicable to the original data with no cut-off. This benchmark is performed over different combinations of background noise Gaussian, Poisson and Uniform and signals Gaussian, Poisson, Uniform, Chirps and Sine waves. Results show performance of EPR to be comparable to that of PC, providing a new approach for significant improvements in efficiency with essentially no loss of sensitivity, relevant to the present era of Big Data observatories.
ER -