Turtles All the Way Down: Parameter Sensitivity in Metaheuristic Hyperparameter Optimisation for Machine Learning Models
- 1Computer Science Division, Stellenbosch University, Stellenbosch 7600, South Africa
- 2Department of Industrial Engineering and Computer Science Division, Stellenbosch University, Stellenbosch 7600, South Africa
Abstract
Metaheuristic algorithms, particularly particle swarm optimisation (PSO) and differential evolution (DE), are widely used for hyperparameter optimisation of machine learning models, yet metaheuristics are in turn sensitive to their own control parameters. How much this recursive dependency affects hyperparameter optimisation outcomes in practice has not been established. Four metaheuristics, including the self-adaptive variants self-adaptive differential evolution (jDE) and self-adaptive particle swarm optimisation (SAPSO), are evaluated across ten datasets and four machine learning models, alongside four modern adaptive methods and three parameter-free baselines. Meta-parameter choices are shown to cause accuracy variations of up to 3.49%. Neither the classical self-adaptive variants nor the modern methods reduce the sensitivity. No population-based metaheuristic significantly outperforms a properly configured grid search, and grid search attains a near-best mean rank at roughly an order of magnitude fewer function evaluations. The metaheuristics are the stronger optimisers of the cross-validation objective. The additional gains do not transfer to test data. The pattern is consistent with oversearching. Simpler methods are therefore preferable for small tabular classification with low-dimensional hyperparameter spaces.
Related
- Sensitivity Analysis for Trials with Irregular and Informative Assessment Times: Marginal Mean Models and Sensitivity Parameter Selection
- Sensitivity-Guided Parameter Estimation of Bubbly Liquid via a Nonlinear Wave Equation and Physics-InformedNeuralNetworks
- Kernelized Parameter Sensitivity Atlases for Amortized Conditional Interpretability
- A Distributional Optimisation Perspective on Combining Models in Deep Learning
- Parameter estimation in Conditional Sequential Monte Carlo algorithms through Particle Learning
BibTeX
@article{anderson2026turtles,
title = {{Turtles All the Way Down: Parameter Sensitivity in Metaheuristic Hyperparameter Optimisation for Machine Learning Models}},
author = {Kian Anderson and Andries Engelbrecht},
journal = {Mathematics},
year = {2026},
month = oct,
doi = {10.3390/math14213956},
url = {https://www.mdpi.com/2227-7390/14/21/3956},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Turtles All the Way Down: Parameter Sensitivity in Metaheuristic Hyperparameter Optimisation for Machine Learning Models
AU - Kian Anderson
AU - Andries Engelbrecht
PY - 2026
DA - 2026/10/29
JO - Mathematics
DO - 10.3390/math14213956
UR - https://www.mdpi.com/2227-7390/14/21/3956
AB - Metaheuristic algorithms, particularly particle swarm optimisation (PSO) and differential evolution (DE), are widely used for hyperparameter optimisation of machine learning models, yet metaheuristics are in turn sensitive to their own control parameters. How much this recursive dependency affects hyperparameter optimisation outcomes in practice has not been established. Four metaheuristics, including the self-adaptive variants self-adaptive differential evolution (jDE) and self-adaptive particle swarm optimisation (SAPSO), are evaluated across ten datasets and four machine learning models, alongside four modern adaptive methods and three parameter-free baselines. Meta-parameter choices are shown to cause accuracy variations of up to 3.49%. Neither the classical self-adaptive variants nor the modern methods reduce the sensitivity. No population-based metaheuristic significantly outperforms a properly configured grid search, and grid search attains a near-best mean rank at roughly an order of magnitude fewer function evaluations. The metaheuristics are the stronger optimisers of the cross-validation objective. The additional gains do not transfer to test data. The pattern is consistent with oversearching. Simpler methods are therefore preferable for small tabular classification with low-dimensional hyperparameter spaces.
SN - 2227-7390
ER -