Cross-validated feature selection using least-squares support vector machines for binary classification
Abstract
Support vector machines (SVMs) are widely used for binary classification. However, irrelevant or redundant features can degrade classification accuracy and model interpretability, making feature selection important for SVMs. Most SVM feature selection methods rely on sequential search or regularization and do not directly identify the best feature subset. Mixed-integer optimization (MIO) is a direct approach to best subset selection. Cross-validation (CV) is broadly applicable to feature selection because it evaluates predictive performance on validation data without model assumptions. Although Purcaru et al. (2026) proposed an MIO formulation for CV-based linear-SVM feature selection (SVM-CV), this method suffers from high computational cost. We develop computationally efficient CV-based MIO formulations for feature selection in linear SVM classification by using least squares support vector machines (LS-SVMs). We formulate the problem as a bilevel MIO problem, where the lower-level problem trains LS-SVM classifiers and the upper-level problem selects features based on validation sets. We then reformulate it as a single-level MIO problem by exploiting the optimality conditions of LS-SVM. We propose two formulations: a hinge-loss-based validation model and a least-squares-based validation model. We compare our methods with SVM-CV and other SVM-based and logistic-regression-based methods through simulation experiments on synthetic data. The hinge-loss-based validation model attained high feature selection accuracy under the high-noise and moderate-noise conditions, particularly with more candidate features, and the least-squares-based validation model performed well in feature selection under the low-noise condition at the lower correlation level. Furthermore, our two models achieved substantially faster computation than the existing SVM-CV model.
Related
- Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines
- Support Vector Machines for Classification, Regression, and Outlier Detection: A Tutorial
- Generalization in Nonlinear Least Squares via Learned Feature Geometry
- Information-Geometric Selection for Validation-Guided Ensembles of Extreme Learning Machines: The LEA-EELM Framework
- Mean-Field Oscillator Ising Machines: Gradient Flows and Classification of Limit Solutions
BibTeX
@misc{mori2026crossvalidated,
title = {{Cross-validated feature selection using least-squares support vector machines for binary classification}},
author = {Masaharu Mori and Shunnosuke Ikeda and Ryuta Tamura and Yuichi Takano and Ryuhei Miyashiro},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.07089},
eprint = {2605.07089},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.07089},
}
RIS
TY - UNPB
TI - Cross-validated feature selection using least-squares support vector machines for binary classification
AU - Masaharu Mori
AU - Shunnosuke Ikeda
AU - Ryuta Tamura
AU - Yuichi Takano
AU - Ryuhei Miyashiro
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2605.07089
UR - https://arxiv.org/abs/2605.07089
AB - Support vector machines (SVMs) are widely used for binary classification. However, irrelevant or redundant features can degrade classification accuracy and model interpretability, making feature selection important for SVMs. Most SVM feature selection methods rely on sequential search or regularization and do not directly identify the best feature subset. Mixed-integer optimization (MIO) is a direct approach to best subset selection. Cross-validation (CV) is broadly applicable to feature selection because it evaluates predictive performance on validation data without model assumptions. Although Purcaru et al. (2026) proposed an MIO formulation for CV-based linear-SVM feature selection (SVM-CV), this method suffers from high computational cost. We develop computationally efficient CV-based MIO formulations for feature selection in linear SVM classification by using least squares support vector machines (LS-SVMs). We formulate the problem as a bilevel MIO problem, where the lower-level problem trains LS-SVM classifiers and the upper-level problem selects features based on validation sets. We then reformulate it as a single-level MIO problem by exploiting the optimality conditions of LS-SVM. We propose two formulations: a hinge-loss-based validation model and a least-squares-based validation model. We compare our methods with SVM-CV and other SVM-based and logistic-regression-based methods through simulation experiments on synthetic data. The hinge-loss-based validation model attained high feature selection accuracy under the high-noise and moderate-noise conditions, particularly with more candidate features, and the least-squares-based validation model performed well in feature selection under the low-noise condition at the lower correlation level. Furthermore, our two models achieved substantially faster computation than the existing SVM-CV model.
ER -