Geometry-Aware Block Selection for Sparse Variational Gaussian Processes
Abstract
Sparse variational Gaussian processes (SVGPs) are a widely used, principled approach to scalable Gaussian process modelling. Their success stems from their theoretically interpretable structure: they provide a lower bound on the marginal likelihood of the full GP while allowing scal- able stochastic optimization. Recent work has explored modifications of the posterior conditional distribution and has shown that such extensions can improve upon this well-established framework. In particular, introducing a block-diagonal structured scal- ing matrix leads to a tighter lower bound on the GP marginal likelihood. We investigate whether selecting blocks that tighten the variational objective improves test RMSE and NLL at a fixed inducing-point budget. Since the scaling matrix contributes to the variational lower bound through an upper bound on the log-determinant of the full residual covariance matrix, our work makes the following contributions. First, we introduce a simple, scalable, geometry-aware algorithm, with explicit computational complexity and low computational overhead, that recovers a large fraction of this log- determinant contribution by selecting an appropriate partition of the data. Second, we show how to train these models using a doubly stochastic optimization framework. Finally, we investigate how sparse GP methods based on alternative conditional posterior structures can be used to learn better hyperparameters and inducing points for SVGP.
Related
- Scalable Sparse Autoregressive Variational Inference for Non-Gaussian Spatial Fields
- Adaptive multi-resolution Gaussian processes: Scalable exact inference with naturally data-sparse covariance matrices
- Penalized Likelihood Inference in Highly Multivariate Sparse Spectral Graphical Gaussian Processes
- Variational Splines and Generalized Gaussian Processes
- On Basis Function Selection for Sparse Gaussian Process Regression
BibTeX
@misc{hoxha2026geometryaware,
title = {{Geometry-Aware Block Selection for Sparse Variational Gaussian Processes}},
author = {Matteo Hoxha},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.28256},
eprint = {2610.28256},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28256},
}
RIS
TY - UNPB
TI - Geometry-Aware Block Selection for Sparse Variational Gaussian Processes
AU - Matteo Hoxha
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.28256
UR - https://arxiv.org/abs/2610.28256
AB - Sparse variational Gaussian processes (SVGPs) are a widely used, principled approach to scalable Gaussian process modelling. Their success stems from their theoretically interpretable structure: they provide a lower bound on the marginal likelihood of the full GP while allowing scal- able stochastic optimization. Recent work has explored modifications of the posterior conditional distribution and has shown that such extensions can improve upon this well-established framework. In particular, introducing a block-diagonal structured scal- ing matrix leads to a tighter lower bound on the GP marginal likelihood. We investigate whether selecting blocks that tighten the variational objective improves test RMSE and NLL at a fixed inducing-point budget. Since the scaling matrix contributes to the variational lower bound through an upper bound on the log-determinant of the full residual covariance matrix, our work makes the following contributions. First, we introduce a simple, scalable, geometry-aware algorithm, with explicit computational complexity and low computational overhead, that recovers a large fraction of this log- determinant contribution by selecting an appropriate partition of the data. Second, we show how to train these models using a doubly stochastic optimization framework. Finally, we investigate how sparse GP methods based on alternative conditional posterior structures can be used to learn better hyperparameters and inducing points for SVGP.
ER -