Joint Estimation and Feature Selection in Overspecified Finite Mixture of Regressions
Abstract
We investigate joint estimation and feature selection in sparse finite mixtures of regressions with an overspecified number of mixture components or mixture order. More specifically, given a conservative fixed upper bound on the unknown mixture order, we deliberately overspecify the order and propose a double-penalized likelihood approach for simultaneous parameter estimation and feature selection. We establish consistent recovery of the sparsity structure of the underlying mixing measure in terms of its regression coefficients, showing that the true sparsity of the mixture regression can be consistently identified even under order over-specification. The proposed methodology incorporates two complementary penalties: the first penalizes small (near-zero) mixing proportions, inducing an asymptotic Voronoi-type partition of the estimated regression vectors into clusters associated with the true sparse regression vectors; the second penalizes regression coefficients, resulting in sparsity through the selection of relevant features. Although the sparsity penalty is applied to individual regression coefficients, the combined effect of the two penalties yields an asymptotic group feature selection property where the same features tend to be selected or not selected within each Voronoi cluster. We develop a modified EM algorithm combined with proximal gradient for numerical implementations. Extensive simulations and a real data application demonstrate the effectiveness of the proposed approach.
Related
- Feature Space Selection and Heterogeneous Effect Estimation for Blood-Brain Barrier Permeability: A Random Forest to the Generalized Random Forest Pipeline
- Sandwich Variance Estimation in Scale Mixture of Skew-Normal Linear Mixed Models.
- Brown's loss function and estimation of the scale parameter and the entropy in scale mixture models with unknown location
- Computer code validation via mixture model estimation
- Mixture-based Nonparametric Estimation of Spatial Covariance Functions with Applications to HIV Key Population Size Estimation across Sub-Saharan Africa
BibTeX
@misc{khalili2026joint,
title = {{Joint Estimation and Feature Selection in Overspecified Finite Mixture of Regressions}},
author = {Abbas Khalili and Pengqi Liu and Huy Nguyen and Nhat Ho},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.22851},
eprint = {2610.22851},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.22851},
}
RIS
TY - UNPB
TI - Joint Estimation and Feature Selection in Overspecified Finite Mixture of Regressions
AU - Abbas Khalili
AU - Pengqi Liu
AU - Huy Nguyen
AU - Nhat Ho
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2610.22851
UR - https://arxiv.org/abs/2610.22851
AB - We investigate joint estimation and feature selection in sparse finite mixtures of regressions with an overspecified number of mixture components or mixture order. More specifically, given a conservative fixed upper bound on the unknown mixture order, we deliberately overspecify the order and propose a double-penalized likelihood approach for simultaneous parameter estimation and feature selection. We establish consistent recovery of the sparsity structure of the underlying mixing measure in terms of its regression coefficients, showing that the true sparsity of the mixture regression can be consistently identified even under order over-specification. The proposed methodology incorporates two complementary penalties: the first penalizes small (near-zero) mixing proportions, inducing an asymptotic Voronoi-type partition of the estimated regression vectors into clusters associated with the true sparse regression vectors; the second penalizes regression coefficients, resulting in sparsity through the selection of relevant features. Although the sparsity penalty is applied to individual regression coefficients, the combined effect of the two penalties yields an asymptotic group feature selection property where the same features tend to be selected or not selected within each Voronoi cluster. We develop a modified EM algorithm combined with proximal gradient for numerical implementations. Extensive simulations and a real data application demonstrate the effectiveness of the proposed approach.
ER -