Sparse Bayesian Neural Networks via Sticky Piecewise Deterministic Markov Processes
Abstract
There is growing evidence that the performance of a neural network depends largely on a small subset of its weights, which are typically identified by training a dense network and pruning it afterwards. Within a Bayesian framework, pruning can be directly incorporated through a spike-and-slab prior, providing a proper treatment of the uncertainties involved, both with respect to pruning and to parameter uncertainty. In Bayesian neural networks (BNNs), sampling by MCMC is notoriously hard. We demonstrate that modern Monte Carlo methods based on piecewise deterministic Markov processes (PDMPs) can be used to sample posteriors of BNNs asymptotically exact whenever numerically constructed rate bounds hold. Spike-and-slab priors can be incorporated through sticky versions of the PDMPs, setting weights exactly to zero during sampling. We examine the predictive performance and convergence of the samplers, and show that sparse BNNs sampled this way can be more robust to input noise than dense ones.
Related
- Accelerating Bayesian Variable Selection using Piecewise Deterministic Markov Processes
- SelectPFN: Amortized Bayesian Variable Selection for Sparse Nonlinear Regression with Prior-Data Fitted Networks
- Fraud Detection via Bayesian Positive-Unlabeled Learning with Gaussian Processes and Multilayer Networks
- Piecewise deterministic sampling for constrained distributions
- The sharp CFL condition of the piecewise constant sparse grid discontinuous Galerkin method for high-dimensional transport equations
BibTeX
@misc{arnstad2026sparse,
title = {{Sparse Bayesian Neural Networks via Sticky Piecewise Deterministic Markov Processes}},
author = {August Arnstad and Geir Olve Storvik and Filippo Pagani and Leiv Rønneberg},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.32753},
eprint = {2610.32753},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.32753},
}
RIS
TY - UNPB
TI - Sparse Bayesian Neural Networks via Sticky Piecewise Deterministic Markov Processes
AU - August Arnstad
AU - Geir Olve Storvik
AU - Filippo Pagani
AU - Leiv Rønneberg
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2610.32753
UR - https://arxiv.org/abs/2610.32753
AB - There is growing evidence that the performance of a neural network depends largely on a small subset of its weights, which are typically identified by training a dense network and pruning it afterwards. Within a Bayesian framework, pruning can be directly incorporated through a spike-and-slab prior, providing a proper treatment of the uncertainties involved, both with respect to pruning and to parameter uncertainty. In Bayesian neural networks (BNNs), sampling by MCMC is notoriously hard. We demonstrate that modern Monte Carlo methods based on piecewise deterministic Markov processes (PDMPs) can be used to sample posteriors of BNNs asymptotically exact whenever numerically constructed rate bounds hold. Spike-and-slab priors can be incorporated through sticky versions of the PDMPs, setting weights exactly to zero during sampling. We examine the predictive performance and convergence of the samplers, and show that sparse BNNs sampled this way can be more robust to input noise than dense ones.
ER -