Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts
5 affiliations
- 1ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems, School of Mathematical Sciences, Queensland University of Technology, GPO Box 2434, Brisbane, QLD, 4001, Australia
- 2Department of Mathematical and Physical Sciences, School of Computing, Engineering, and Mathematical Sciences, La Trobe University, Bundoora, VIC, 3086, Australia
- 3Institute of Mathematics for Industry, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka, 819-0395, Japan
- 4IRT SystemX, 2 Boulevard Thomas Gobert, 91120, Palaiseau, France
- 5School of Mathematics and Physics, The University of Queensland, St Lucia, Brisbane, QLD, 4072, Australia
Abstract
We study Lasso estimation for high-dimensional softmax-gated Gaussian mixture-of-experts (SGMoE) regression models, where the predictor dimension P may exceed the sample size N . Under bounded parameters, we establish a non-asymptotic Kullback–Leibler (KL) oracle inequality with an explicit sufficient regularization level and explicit dependence on N , P , response dimension Q , number of experts K , and boundedness constants. We derive sparse well-specified and approximation-space rates, showing how deterministic SGMoE approximation error transfers into statistical KL-risk bounds. The theory extends to random covariates and, under misspecification, controls the best Gaussian SGMoE approximation in the candidate class. Reproducible experiments validate KL decay and density recovery, compare Lasso with SCAD when P>N P > N , evaluate model-selection criteria, handle multivariate covariance structures, and assess high-dimensional biological benchmarks relevant to heterogeneous genotype–phenotype mapping, omics-to-trait prediction, and AI-enabled virtual-cell modelling in systems biology.
Related
- Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts
- Mixture of Censored Partially Linear Experts Models for Symmetric or Asymmetric Data with or without Outlying Observations: A novel approach with two-piece scale-mixture of normal distributions
- The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts
- A Mixture-of-Experts Framework for Multi-Stream Quickest Change Detection
- When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts
BibTeX
@article{nguyen2026nonasymptotic,
title = {{Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts}},
author = {TrungTin Nguyen and Hien D. Nguyen and Faïcel Chamroukhi and Geoffrey John McLachlan},
journal = {Annals of the Institute of Statistical Mathematics},
year = {2026},
month = oct,
doi = {10.1007/s10463-026-01007-5},
url = {https://doi.org/10.1007/s10463-026-01007-5},
note = {CC BY},
}
RIS
TY - JOUR
TI - Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts
AU - TrungTin Nguyen
AU - Hien D. Nguyen
AU - Faïcel Chamroukhi
AU - Geoffrey John McLachlan
PY - 2026
DA - 2026/10/16
JO - Annals of the Institute of Statistical Mathematics
DO - 10.1007/s10463-026-01007-5
UR - https://doi.org/10.1007/s10463-026-01007-5
AB - We study Lasso estimation for high-dimensional softmax-gated Gaussian mixture-of-experts (SGMoE) regression models, where the predictor dimension P may exceed the sample size N . Under bounded parameters, we establish a non-asymptotic Kullback–Leibler (KL) oracle inequality with an explicit sufficient regularization level and explicit dependence on N , P , response dimension Q , number of experts K , and boundedness constants. We derive sparse well-specified and approximation-space rates, showing how deterministic SGMoE approximation error transfers into statistical KL-risk bounds. The theory extends to random covariates and, under misspecification, controls the best Gaussian SGMoE approximation in the candidate class. Reproducible experiments validate KL decay and density recovery, compare Lasso with SCAD when P>N P > N , evaluate model-selection criteria, handle multivariate covariance structures, and assess high-dimensional biological benchmarks relevant to heterogeneous genotype–phenotype mapping, omics-to-trait prediction, and AI-enabled virtual-cell modelling in systems biology.
SN - 0020-3157
SN - 1572-9052
ER -