Theoretical and Practical Advances in Hierarchical Bayesian Deep Learning Methodology
Abstract
Reliable uncertainty quantification remains an important challenge in AI. Hierarchical Bayesian methods provide a principled framework for this by representing unknown quantities through posterior distributions. Joint inference of neural-network parameters and prior hyperparameters is challenging for conventional Markov chain Monte Carlo (MCMC) methods due to strong dependencies. In this work, we introduce the multilevel particle marginal Metropolis-Hastings (MLPMMH) method, which combines pseudo-marginal inference for the hyperparameters with coupled sequential Monte Carlo (SMC) across hierarchy of network-width truncations for conditional parameter inference. The hierarchy is induced by a trace-class neural network (TNN) prior with width-dependent decay, motivated empirically by the compressible structure observed in trained neural-network parameters. Under standard multilevel coupling assumptions, we establish that MLPMMH achieves mean-square error (MSE) 𝒪(ε2) at expected computational cost 𝒪(ε−2), and verify the predicted rate on a Bayesian regression problem. To address larger-scale applications, we further introduce S-MLPMMH, a practical approximation that replaces the exact sampling components with a PCA-based hierarchy and scalable Bayesian Monte Carlo. Experiments on MNIST demonstrate competitive predictive performance and uncertainty quantification while exploiting the parallel structure of the approximation. Software has been made publicly available at https://github.com/submission-code-anon/mlpmmh-anon.
Related
- Bayesian Hierarchical Models for Longitudinal Meta-Analysis in Sports Science: A Practical Tutorial Using brms
- Some theoretical and practical results on noisy signals recovery
- A novel Arctan-based model: Bayesian analysis, progressive censoring, and practical applications
- Theoretical and Computational Advances in Birkhoff Pseudospectral Transcription for Aerospace Optimal Control
- Frequentist and Bayesian population pharmacometrics: similarities, differences, and practical implications
BibTeX
@misc{chada2026theoretical,
title = {{Theoretical and Practical Advances in Hierarchical Bayesian Deep Learning Methodology}},
author = {Neil K. Chada and Simon Cotter and Kody J. H. Law and Xinzhu Liang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.19074},
eprint = {2610.19074},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.19074},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - Theoretical and Practical Advances in Hierarchical Bayesian Deep Learning Methodology
AU - Neil K. Chada
AU - Simon Cotter
AU - Kody J. H. Law
AU - Xinzhu Liang
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2610.19074
UR - https://arxiv.org/abs/2610.19074
AB - Reliable uncertainty quantification remains an important challenge in AI. Hierarchical Bayesian methods provide a principled framework for this by representing unknown quantities through posterior distributions. Joint inference of neural-network parameters and prior hyperparameters is challenging for conventional Markov chain Monte Carlo (MCMC) methods due to strong dependencies. In this work, we introduce the multilevel particle marginal Metropolis-Hastings (MLPMMH) method, which combines pseudo-marginal inference for the hyperparameters with coupled sequential Monte Carlo (SMC) across hierarchy of network-width truncations for conditional parameter inference. The hierarchy is induced by a trace-class neural network (TNN) prior with width-dependent decay, motivated empirically by the compressible structure observed in trained neural-network parameters. Under standard multilevel coupling assumptions, we establish that MLPMMH achieves mean-square error (MSE) 𝒪(ε^2) at expected computational cost 𝒪(ε^−2), and verify the predicted rate on a Bayesian regression problem. To address larger-scale applications, we further introduce S-MLPMMH, a practical approximation that replaces the exact sampling components with a PCA-based hierarchy and scalable Bayesian Monte Carlo. Experiments on MNIST demonstrate competitive predictive performance and uncertainty quantification while exploiting the parallel structure of the approximation. Software has been made publicly available at https://github.com/submission-code-anon/mlpmmh-anon.
ER -