Continuous-depth Deep Gaussian Processes
Abstract
Deep Gaussian processes (DGPs) are appealing Bayesian regression models, but their standard discrete-depth parameterization often becomes harder to optimize as depth grows. We argue that depth is better treated as a latent continuous evolution than as a long discrete stack. We therefore propose a continuous-depth formulation in which the latent representation follows a stochastic differential equation with a depth-indexed drift family equipped with a GP-based prior. This reframes the core design choice as the prior placed on that family, rather than adopting a purely neural continuous-depth parameterization as in neural ODE or neural SDE models. We study two instantiations of this idea: CDGP, a direct GP prior over the state--depth drift, and FlowDGP, a flow-evolved prior that captures richer depth dependence at lower training cost. To motivate the redesign, we analyze how the single-sample DGP objective can become increasingly sensitive under repeated layer composition, while residual DGP is a useful first repair that still remains a finite discrete construction. We also give pathwise sensitivity analysis for the continuous-depth formulation. Experiments on synthetic, benchmark, and large-scale regression tasks show that residual DGP improves over plain DGP, while CDGP and FlowDGP provide a stronger overall story across the regimes we study.
Related
- Bayesian Estimation of Continuous Brain Connectivity with Log-Gaussian Cox Processes
- Continuous mixtures of Gaussian processes as models for spatial extremes
- Wavelet compression for sampling deep Gaussian processes
- Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation
- Development of approximate scheduling-adaptive controllers for multi-products continuous chemical processes using deep learning techniques and model predictive control
BibTeX
@misc{li2026continuousdepth,
title = {{Continuous-depth Deep Gaussian Processes}},
author = {Ying Li and Sanyou Wu and Zi Yang and Qiaochu Xu and Yuhao Liu and Zhidi Lin and Michael Minyi Zhang and Petar Djuric and Alexandre H. Thiery},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.30890},
eprint = {2610.30890},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.30890},
}
RIS
TY - UNPB
TI - Continuous-depth Deep Gaussian Processes
AU - Ying Li
AU - Sanyou Wu
AU - Zi Yang
AU - Qiaochu Xu
AU - Yuhao Liu
AU - Zhidi Lin
AU - Michael Minyi Zhang
AU - Petar Djuric
AU - Alexandre H. Thiery
PY - 2026
DA - 2026/10/22
PB - arXiv
DO - 10.48550/arxiv.2610.30890
UR - https://arxiv.org/abs/2610.30890
AB - Deep Gaussian processes (DGPs) are appealing Bayesian regression models, but their standard discrete-depth parameterization often becomes harder to optimize as depth grows. We argue that depth is better treated as a latent continuous evolution than as a long discrete stack. We therefore propose a continuous-depth formulation in which the latent representation follows a stochastic differential equation with a depth-indexed drift family equipped with a GP-based prior. This reframes the core design choice as the prior placed on that family, rather than adopting a purely neural continuous-depth parameterization as in neural ODE or neural SDE models. We study two instantiations of this idea: CDGP, a direct GP prior over the state--depth drift, and FlowDGP, a flow-evolved prior that captures richer depth dependence at lower training cost. To motivate the redesign, we analyze how the single-sample DGP objective can become increasingly sensitive under repeated layer composition, while residual DGP is a useful first repair that still remains a finite discrete construction. We also give pathwise sensitivity analysis for the continuous-depth formulation. Experiments on synthetic, benchmark, and large-scale regression tasks show that residual DGP improves over plain DGP, while CDGP and FlowDGP provide a stronger overall story across the regimes we study.
ER -