Covariate-dependent Nonparametric g-modeling for regression via infinite Mixture-of-Expertizing class
Abstract
Empirical Bayes g-modeling captures unit-level heterogeneity by estimating a latent prior distribution from observed data. In the existing formulations, however, the prior is shared by all units. In this paper, we develop a covariate-dependent g-modeling framework for regression in which the entire prior distribution of the regression coefficients is allowed to depend on covariates. We formulate the estimation of the prior as nonparametric maximum likelihood estimation (NPMLE) of the covariate-dependent prior, and show that the unrestricted problem is ill-posed. To resolve this, we introduce the infinite Mixture-of-Expertizing class of conditional priors, under which the NPMLE is precisely a softmax-gated Mixture of Experts (MoE) whose number of experts is not fixed in advance but is determined by the data. Building on a first-order optimality condition, we propose two exemplar-based estimation algorithms that select experts automatically, together with a post-hoc aggregation of experts for interpretation. On the theoretical side, we show that every conditional prior in the class is Lipschitz continuous in the covariates, and that aggregated softmax gates can approximate any continuous gate function. The effectiveness of the proposed NPMLE is shown through application to synthetic datasets and real datasets.
Related
- Optimal Neural Estimation for High-Dimensional Nonparametric Regression under Multisource Covariate Shift
- Reconciling Interpretability with Covariate-Dependent Shape Flexibility in Penalized Transformation Models for Distributional Regression
- Covariate-dependent Joint Modeling of Multivariate Ordinal Preferences and Its Connections with Comparison Models
- Nonparametric Intensity Estimation for Covariate-Driven Poisson Processes Using Deep Learning
- Conformalized Quantile Regression and Minimax Limits of Fixed-Score Calibration under Known Covariate Shift
BibTeX
@misc{okazaki2026covariatedependent,
title = {{Covariate-dependent Nonparametric $g$-modeling for regression via infinite Mixture-of-Expertizing class}},
author = {Akira Okazaki and Keisuke Yano},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.08321},
eprint = {2610.08321},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.08321},
}
RIS
TY - UNPB
TI - Covariate-dependent Nonparametric g-modeling for regression via infinite Mixture-of-Expertizing class
AU - Akira Okazaki
AU - Keisuke Yano
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.08321
UR - https://arxiv.org/abs/2610.08321
AB - Empirical Bayes g-modeling captures unit-level heterogeneity by estimating a latent prior distribution from observed data. In the existing formulations, however, the prior is shared by all units. In this paper, we develop a covariate-dependent g-modeling framework for regression in which the entire prior distribution of the regression coefficients is allowed to depend on covariates. We formulate the estimation of the prior as nonparametric maximum likelihood estimation (NPMLE) of the covariate-dependent prior, and show that the unrestricted problem is ill-posed. To resolve this, we introduce the infinite Mixture-of-Expertizing class of conditional priors, under which the NPMLE is precisely a softmax-gated Mixture of Experts (MoE) whose number of experts is not fixed in advance but is determined by the data. Building on a first-order optimality condition, we propose two exemplar-based estimation algorithms that select experts automatically, together with a post-hoc aggregation of experts for interpretation. On the theoretical side, we show that every conditional prior in the class is Lipschitz continuous in the covariates, and that aggregated softmax gates can approximate any continuous gate function. The effectiveness of the proposed NPMLE is shown through application to synthetic datasets and real datasets.
ER -