Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling
Abstract
Black-box machine learning models increasingly deliver strong predictions, but extracting useful information from them, such as a set of important features, remains challenging. Existing model-agnostic methods primarily estimate feature importance or conduct inference on it rather than directly selecting features, whereas many feature selection methods are model-specific or rely on the model-X assumption. We introduce LOCO-guided Adaptive Minipatch Sampling (LAMPS), a model-agnostic ensemble framework that uses any black-box regression algorithm as its base learner to select features important for predicting the response. The base learner need only produce predictions and need not perform feature selection itself. LAMPS operates within a minipatch ensemble framework that subsamples both observations and features, allowing leave-one-covariate-out (LOCO) feature importance scores to be easily computed. It adaptively concentrates minipatch sampling on features with high LOCO scores while maintaining exploration. The resulting sampling probabilities rapidly separate signal from noise features after a few iterations, enabling selection through simple thresholding. We establish that LAMPS achieves exact feature selection in high-dimensional settings, provided that the base predictive models are sufficiently well trained on average. Extensive experiments on synthetic and real data show that LAMPS outperforms state-of-the-art feature selection methods, with particularly strong performance in the presence of correlated features.
Related
- LOCO-AdaMP: Built-in LOCO Inference for Adaptive Minipatch Ensembles with Enhanced Prediction
- ARISE: An adaptive residual-informed stability ensemble for feature selection in small-sample biomedical omics
- Multicollinearity-agnostic feature screening for non-Euclidean responses: a factor adjusted approach
- When n is not enough: Assessing the interplay of spatial sampling error and sample selection in abundance estimation
- Linear-Time Multivariate Unsupervised Feature Selection Using Nyström-Based Entropy Approximation
BibTeX
@misc{liu2026modelagnostic,
title = {{Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling}},
author = {Xuhui Liu and Lili Zheng},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.24126},
eprint = {2609.24126},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.24126},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling
AU - Xuhui Liu
AU - Lili Zheng
PY - 2026
DA - 2026/09/21
PB - arXiv
DO - 10.48550/arxiv.2609.24126
UR - https://arxiv.org/abs/2609.24126
AB - Black-box machine learning models increasingly deliver strong predictions, but extracting useful information from them, such as a set of important features, remains challenging. Existing model-agnostic methods primarily estimate feature importance or conduct inference on it rather than directly selecting features, whereas many feature selection methods are model-specific or rely on the model-X assumption. We introduce LOCO-guided Adaptive Minipatch Sampling (LAMPS), a model-agnostic ensemble framework that uses any black-box regression algorithm as its base learner to select features important for predicting the response. The base learner need only produce predictions and need not perform feature selection itself. LAMPS operates within a minipatch ensemble framework that subsamples both observations and features, allowing leave-one-covariate-out (LOCO) feature importance scores to be easily computed. It adaptively concentrates minipatch sampling on features with high LOCO scores while maintaining exploration. The resulting sampling probabilities rapidly separate signal from noise features after a few iterations, enabling selection through simple thresholding. We establish that LAMPS achieves exact feature selection in high-dimensional settings, provided that the base predictive models are sufficiently well trained on average. Extensive experiments on synthetic and real data show that LAMPS outperforms state-of-the-art feature selection methods, with particularly strong performance in the presence of correlated features.
ER -