Approximating Bayesian leave-one-group-out cross-validation
Abstract
When data are grouped, hierarchical or multilevel models are commonly used to account for group-level variation with group-specific parameters. Leave-one-group-out cross-validation (LOGO-CV) is a suitable tool for evaluating predictive performance for new groups, providing an estimator of the expected log predictive density (elpd). Brute-force LOGO-CV requires one model refit per held-out group, often using computationally expensive inference algorithms such as MCMC. This is costly, particularly for large numbers of groups or complex model structures. Commonly used importance sampling approximations, intended to reduce this cost, tend to fail because the group-specific parameters of the held-out group must be integrated out. We identify two key challenges in LOGO-CV elpd estimation: approximating the LOGO posterior and computing the grouped marginal likelihood. We compare 11 strategies, including 5 newly proposed, to address them. Among others, we combine Pareto-smoothed importance sampling or adaptive importance sampling with integration techniques such as Laplace approximation, adaptive Gauss-Hermite quadrature, and bridge sampling. We evaluate these strategies in both simulation experiments and real-world case studies, which show that marginalising over the group-specific parameters substantially improves the reliability of the importance sampling approaches.
Related
- Deterministic Leave-One-Cluster-Out Cross-Validation for Multilevel Bayesian Structural Equation Models
- Testing Group Differences in Network Models using Bayesian Model Selection: A Tutorial
- Bayesian inference of risk differences for multi-group bilateral data
- Bayesian Group Testing Regression with a Shape-Free Dilution Curve
- Bayesian calibration with model discrepancy of a two-group interfacial area transport model for upward bubbly flow in a square duct
BibTeX
@misc{riha2026approximating,
title = {{Approximating Bayesian leave-one-group-out cross-validation}},
author = {Anna Elisabeth Riha and Svenja Jedhoff and Paul-Christian Bürkner and Aki Vehtari},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.05713},
eprint = {2609.05713},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.05713},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Approximating Bayesian leave-one-group-out cross-validation
AU - Anna Elisabeth Riha
AU - Svenja Jedhoff
AU - Paul-Christian Bürkner
AU - Aki Vehtari
PY - 2026
DA - 2026/09/04
PB - arXiv
DO - 10.48550/arxiv.2609.05713
UR - https://arxiv.org/abs/2609.05713
AB - When data are grouped, hierarchical or multilevel models are commonly used to account for group-level variation with group-specific parameters. Leave-one-group-out cross-validation (LOGO-CV) is a suitable tool for evaluating predictive performance for new groups, providing an estimator of the expected log predictive density (elpd). Brute-force LOGO-CV requires one model refit per held-out group, often using computationally expensive inference algorithms such as MCMC. This is costly, particularly for large numbers of groups or complex model structures. Commonly used importance sampling approximations, intended to reduce this cost, tend to fail because the group-specific parameters of the held-out group must be integrated out. We identify two key challenges in LOGO-CV elpd estimation: approximating the LOGO posterior and computing the grouped marginal likelihood. We compare 11 strategies, including 5 newly proposed, to address them. Among others, we combine Pareto-smoothed importance sampling or adaptive importance sampling with integration techniques such as Laplace approximation, adaptive Gauss-Hermite quadrature, and bridge sampling. We evaluate these strategies in both simulation experiments and real-world case studies, which show that marginalising over the group-specific parameters substantially improves the reliability of the importance sampling approaches.
ER -