How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPs

Abstract

Compositional priors describe the generic properties of layered functions in deep Bayesian models, where deep neural networks with random weights are a canonical example. In the wide-network limit, the prior is a Gaussian process (GP) with a depth-dependent kernel, and its behaviour as depth grows has been extensively studied through this kernel. Here, we study another case, where each layer itself is a vector valued Gaussian process, and our aim is similarly to understand the limiting behaviour of the prior as depth grows. Previous GP work has established that for the radial basis function (RBF) kernel and a certain range of bandwidths r, the prior degenerates in the limit, converging to the set of constant functions, which is not useful as a probabilistic model. In this paper we establish several new results. First, we identify a sharp bandwidth threshold r_c(d) = Θ(√d) above which the limit is degenerate, strengthening the earlier bounds. Second, and more importantly, we show that for r below the threshold r_c(d) the prior converges to a limit distribution π_Z̅. We also prove that these distributions are non-degenerate and non-Gaussian, with non-vanishing dependence between coordinates. In contrast to the previously known degenerate regime, deep Gaussian process priors can therefore admit non-trivial limits. Empirically, we verify the threshold across a range of dimensions d, and demonstrate a complex multimodal behaviour of the limit distributions π_Z̅, a regime that becomes increasingly narrow with d and would be hard to identify without knowing the threshold.

The Sharp Ellipticity Threshold for W^1,1-Regularity
Franz Gmeineder
arXiv24 Sep 2026PreprintarXiv licenceFull textAnalysis & PDEsarXiv
The sharp and non-sharp threshold phenomenon in buffered and Go-or-Grow systems
Ruben Taieb, Arnaud Ducrot
HAL20 Sep 2026PreprintCC BY-NC-ND 4.0Full textAnalysis & PDEsHAL
A Sharp Mass Threshold for Fractional Choquard Equations with Lower Hardy-Littlewood-Sobolev and L²-Critical Terms
Yongpeng Chen, Zhipeng Yang
arXiv10 Sep 2026PreprintarXiv licenceFull textAnalysis & PDEsarXiv
A SHARP MASS THRESHOLD FOR BOUNDEDNESS IN A CRITICAL QUASILINEAR CHEMOTAXIS MODEL WITH INDIRECT SIGNAL PRODUCTION: THE RADIALLY SYMMETRIC CASE
Philippe Laurençot, Christoph Walker
HAL31 Aug 2026PreprintCC BY 4.0Full textAnalysis & PDEsHALarXiv
The Sharp C^1,1 Target Threshold for Minimizing Constraint Maps
Yilu Liu
arXiv8 Sep 2026PreprintarXiv licenceFull textAnalysis & PDEsarXiv

Cite

BibTeX RIS

BibTeX

@misc{kozdoba2026deep,
  title = {{How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPs}},
  author = {Mark Kozdoba and Shie Mannor},
  howpublished = {arXiv},
  year = {2026},
  month = nov,
  doi = {10.48550/arxiv.2606.08218},
  eprint = {2606.08218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.08218},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPs
AU  - Mark Kozdoba
AU  - Shie Mannor
PY  - 2026
DA  - 2026/11/03
PB  - arXiv
DO  - 10.48550/arxiv.2606.08218
UR  - https://arxiv.org/abs/2606.08218
AB  - Compositional priors describe the generic properties of layered functions in deep Bayesian models, where deep neural networks with random weights are a canonical example. In the wide-network limit, the prior is a Gaussian process (GP) with a depth-dependent kernel, and its behaviour as depth grows has been extensively studied through this kernel. Here, we study another case, where each layer itself is a vector valued Gaussian process, and our aim is similarly to understand the limiting behaviour of the prior as depth grows. Previous GP work has established that for the radial basis function (RBF) kernel and a certain range of bandwidths r, the prior degenerates in the limit, converging to the set of constant functions, which is not useful as a probabilistic model. In this paper we establish several new results. First, we identify a sharp bandwidth threshold r_c(d) = Θ(√d) above which the limit is degenerate, strengthening the earlier bounds. Second, and more importantly, we show that for r below the threshold r_c(d) the prior converges to a limit distribution π_Z̅. We also prove that these distributions are non-degenerate and non-Gaussian, with non-vanishing dependence between coordinates. In contrast to the previously known degenerate regime, deep Gaussian process priors can therefore admit non-trivial limits. Empirically, we verify the threshold across a range of dimensions d, and demonstrate a complex multimodal behaviour of the limit distributions π_Z̅, a regime that becomes increasingly narrow with d and would be hard to identify without knowing the threshold.
ER  -

Abstract

Related

Cite