Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel
Abstract
Gaussian processes (GPs) are widely used across machine learning, spatial statistics, time-series analysis, optimization, Bayesian statistics, and scientific applications. A central component of a GP model is its kernel, which is typically specified through a parametric family. Among the most widely used choices is the radial basis function (RBF), also known as the squared exponential or Gaussian kernel, owing to its simple form, smoothness, and flexibility. In practice, the kernel parameters are routinely estimated by the maximum likelihood estimators (MLEs), as implemented by standard GP software. Despite this widespread use, the asymptotic behavior of the MLEs remains poorly understood under fixed-domain asymptotics, even for the RBF kernel. The main difficulty arises from the increasingly strong dependence among densely sampled observations and the nonlinear dependence of the covariance matrix on the kernel parameters. In this paper, we address this gap by providing, to the best of our knowledge, the first complete asymptotic characterization of the joint MLE of the spatial variance, lengthscale, and nugget variance under fixed-domain asymptotics. We establish consistency, derive convergence rates for all three parameters, prove joint asymptotic normality, and show that these rates are minimax optimal.
Related
- Marginal maximum likelihood estimation and asymptotic theory for latent variable models in high dimensions
- Finite-Rank Logistic Gaussian Processes with Exact Likelihood for Conditional Density Estimation
- A Non-Asymptotic Gaussian Likelihood Theory for Long-Memory Toeplitz Models
- Asymptotic Theory for Restricted Maximum Likelihood Estimators in Generalized Linear Mixed Models
- Restricted Maximum Likelihood Estimation in Generalized Linear Mixed Models
BibTeX
@misc{qaqish2026sharp,
title = {{Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel}},
author = {Ameer Qaqish and Sudipto Banerjee and Didong Li},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10080},
eprint = {2610.10080},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10080},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel
AU - Ameer Qaqish
AU - Sudipto Banerjee
AU - Didong Li
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.10080
UR - https://arxiv.org/abs/2610.10080
AB - Gaussian processes (GPs) are widely used across machine learning, spatial statistics, time-series analysis, optimization, Bayesian statistics, and scientific applications. A central component of a GP model is its kernel, which is typically specified through a parametric family. Among the most widely used choices is the radial basis function (RBF), also known as the squared exponential or Gaussian kernel, owing to its simple form, smoothness, and flexibility. In practice, the kernel parameters are routinely estimated by the maximum likelihood estimators (MLEs), as implemented by standard GP software. Despite this widespread use, the asymptotic behavior of the MLEs remains poorly understood under fixed-domain asymptotics, even for the RBF kernel. The main difficulty arises from the increasingly strong dependence among densely sampled observations and the nonlinear dependence of the covariance matrix on the kernel parameters. In this paper, we address this gap by providing, to the best of our knowledge, the first complete asymptotic characterization of the joint MLE of the spatial variance, lengthscale, and nugget variance under fixed-domain asymptotics. We establish consistency, derive convergence rates for all three parameters, prove joint asymptotic normality, and show that these rates are minimax optimal.
ER -