A Bayesian Bi-Directional Splitting Framework for Variable Selection in Large Datasets
Abstract
Modern tabular datasets are becoming increasingly large, both in the number of samples and covariates, posing significant challenges for Bayesian variable selection due to the resulting computational burden. While there is extensive literature on scaling Bayesian inference to large numbers of observations or high-dimensional covariate spaces, comparatively little work addresses scalable Bayesian variable selection when both dimensions are large simultaneously in a practical setting. This paper presents a novel Bayesian variable selection framework for efficiently analysing data with a large number of both rows and columns. The proposed framework operates via a divide-and-conquer approach, splitting data into batches along both directions and analysing each batch independently in parallel, after which the results are combined together in a two-phase consensus procedure. Experiments demonstrate the computational gains while retaining strong variable selection performance, successfully identifying relevant signals in noisy, high-dimensional settings despite the loss of information induced by data splitting. Practical guidelines are also provided, including recommendations for tuning parameter choices and effective data partitioning strategies, followed by a real application to an H3N2 influenza dataset.
Related
- Bayesian Variable Selection for High-Dimensional Predictors with Missing Psychometric Outcomes
- Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models
- Refining Relational Event Models: Bayesian Penalization and Variable Selection in REMs
- Whole-Sequence Convergence of Variable-Smoothing Full-Splitting Methods via a Lifted Kurdyka-Łojasiewicz Framework
- Accelerating Bayesian Variable Selection using Piecewise Deterministic Markov Processes
BibTeX
@misc{coats2026bayesian,
title = {{A Bayesian Bi-Directional Splitting Framework for Variable Selection in Large Datasets}},
author = {Aaron Coats and Vinny Davies and Mayetri Gupta},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.20000},
eprint = {2609.20000},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.20000},
}
RIS
TY - UNPB
TI - A Bayesian Bi-Directional Splitting Framework for Variable Selection in Large Datasets
AU - Aaron Coats
AU - Vinny Davies
AU - Mayetri Gupta
PY - 2026
DA - 2026/09/17
PB - arXiv
DO - 10.48550/arxiv.2609.20000
UR - https://arxiv.org/abs/2609.20000
AB - Modern tabular datasets are becoming increasingly large, both in the number of samples and covariates, posing significant challenges for Bayesian variable selection due to the resulting computational burden. While there is extensive literature on scaling Bayesian inference to large numbers of observations or high-dimensional covariate spaces, comparatively little work addresses scalable Bayesian variable selection when both dimensions are large simultaneously in a practical setting. This paper presents a novel Bayesian variable selection framework for efficiently analysing data with a large number of both rows and columns. The proposed framework operates via a divide-and-conquer approach, splitting data into batches along both directions and analysing each batch independently in parallel, after which the results are combined together in a two-phase consensus procedure. Experiments demonstrate the computational gains while retaining strong variable selection performance, successfully identifying relevant signals in noisy, high-dimensional settings despite the loss of information induced by data splitting. Practical guidelines are also provided, including recommendations for tuning parameter choices and effective data partitioning strategies, followed by a real application to an H3N2 influenza dataset.
ER -