Isotropic Gaussian Processes Improve Vanilla Bayesian Optimization in High Dimensions
Abstract
High-dimensional Bayesian optimization (BO) often fits Gaussian process (GP) surrogates from far fewer observations than input dimensions. Modern Vanilla BO can perform well in this regime with dimension-aware priors, initialization, and acquisition optimization, but it typically retains automatic relevance determination (ARD), fitting one lengthscale per input coordinate. We study this modeling choice and propose Iso-BO, a controlled modification that replaces the ARD GP with an isotropic GP using one shared lengthscale while keeping the surrounding BO pipeline matched. For radial kernels, we show that the marginal log likelihood (MLL) depends on the inverse-squared ARD lengthscales only through weighted pairwise distances among the observed inputs. The current design can therefore leave some ARD directions exactly invisible or only weakly constrained by the MLL. Iso-BO removes coordinatewise reweighting and fits a single shared scale instead. Lengthscale-fitting and predictive-density diagnostics show that this finite-data effect appears in practice, including when the data-generating process is anisotropic. Across GP-prior, synthetic, and real-world benchmarks, Iso-BO often improves over matched modern Vanilla BO and remains competitive with the included high-dimensional BO baselines under the tested budgets. Stress tests also show the expected boundary wherein sufficiently strong, learnable anisotropy can favor the more flexible ARD model.
Related
- Goal-Oriented Lower-Tail Calibration of Gaussian Processes for Bayesian Optimization
- Whittle--Matérn metrics and isotropic Gaussian processes on compact metric graphs
- Bayesian Estimation of Continuous Brain Connectivity with Log-Gaussian Cox Processes
- Bayesian Emulation of Multi-fidelity Earth System Modelling Using Hierarchical Gaussian Processes
- Fraud Detection via Bayesian Positive-Unlabeled Learning with Gaussian Processes and Multilayer Networks
BibTeX
@misc{tang2026isotropic,
title = {{Isotropic Gaussian Processes Improve Vanilla Bayesian Optimization in High Dimensions}},
author = {Wei-Ting Tang and Madhav Muthyala and Joel A. Paulson},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.05780},
eprint = {2610.05780},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.05780},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Isotropic Gaussian Processes Improve Vanilla Bayesian Optimization in High Dimensions
AU - Wei-Ting Tang
AU - Madhav Muthyala
AU - Joel A. Paulson
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.05780
UR - https://arxiv.org/abs/2610.05780
AB - High-dimensional Bayesian optimization (BO) often fits Gaussian process (GP) surrogates from far fewer observations than input dimensions. Modern Vanilla BO can perform well in this regime with dimension-aware priors, initialization, and acquisition optimization, but it typically retains automatic relevance determination (ARD), fitting one lengthscale per input coordinate. We study this modeling choice and propose Iso-BO, a controlled modification that replaces the ARD GP with an isotropic GP using one shared lengthscale while keeping the surrounding BO pipeline matched. For radial kernels, we show that the marginal log likelihood (MLL) depends on the inverse-squared ARD lengthscales only through weighted pairwise distances among the observed inputs. The current design can therefore leave some ARD directions exactly invisible or only weakly constrained by the MLL. Iso-BO removes coordinatewise reweighting and fits a single shared scale instead. Lengthscale-fitting and predictive-density diagnostics show that this finite-data effect appears in practice, including when the data-generating process is anisotropic. Across GP-prior, synthetic, and real-world benchmarks, Iso-BO often improves over matched modern Vanilla BO and remains competitive with the included high-dimensional BO baselines under the tested budgets. Stress tests also show the expected boundary wherein sufficiently strong, learnable anisotropy can favor the more flexible ARD model.
ER -