On the Performance Evaluation of Anytime Algorithms in Noisy Optimization
- 1Institute for Information Systems, University of Applied Sciences and Arts Northwestern Switzerland, 4600 Olten, Switzerland
Abstract
A standardized, elaborate and well-argued performance metric for benchmarking anytime algorithms on noisy optimization problems is missing in the current benchmarking frameworks. This research aims at developing such a metric as an artifact by following the Design Science Research methodology. Existing noise handling practices and benchmarking guidelines are reviewed. Benchmarking experiments, including a test suite containing 13 artificial test functions, were conducted with two state-of-the-art algorithms to assess the performance metric. Based on the experimental results and the existing literature, this study concludes that, rather than relying on a single quantitative value, algorithm performance should be assessed through a comprehensive performance report comprising the numerical loss value, the number of function evaluations, wall-clock runtime, and a performance profile illustrating the trajectory of loss values over time. This approach aims to maximize the reliability, expressiveness, and informational value of performance assessments.
Related
- Improving the Last-Iterate Guarantees of Anytime Algorithms for Stochastic Monotone Variational Inequalities
- Performance Evaluation of Metaheuristic Algorithms for Fractional-Order PID Controller Tuning in Non-Linear Systems
- A general recipe for data-adaptive anytime-valid envelopes for iterative algorithms
- A Short Note on Algorithms For The Noisy k-XOR Problem
- Cascading Bandits with Noisy Observations: Algorithms and Regret Bounds
BibTeX
@article{boeh2026performance,
title = {{On the Performance Evaluation of Anytime Algorithms in Noisy Optimization}},
author = {Ramona Boeh and Thomas Hanne},
journal = {Mathematics},
year = {2026},
month = oct,
doi = {10.3390/math14213958},
url = {https://www.mdpi.com/2227-7390/14/21/3958},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - On the Performance Evaluation of Anytime Algorithms in Noisy Optimization
AU - Ramona Boeh
AU - Thomas Hanne
PY - 2026
DA - 2026/10/29
JO - Mathematics
DO - 10.3390/math14213958
UR - https://www.mdpi.com/2227-7390/14/21/3958
AB - A standardized, elaborate and well-argued performance metric for benchmarking anytime algorithms on noisy optimization problems is missing in the current benchmarking frameworks. This research aims at developing such a metric as an artifact by following the Design Science Research methodology. Existing noise handling practices and benchmarking guidelines are reviewed. Benchmarking experiments, including a test suite containing 13 artificial test functions, were conducted with two state-of-the-art algorithms to assess the performance metric. Based on the experimental results and the existing literature, this study concludes that, rather than relying on a single quantitative value, algorithm performance should be assessed through a comprehensive performance report comprising the numerical loss value, the number of function evaluations, wall-clock runtime, and a performance profile illustrating the trajectory of loss values over time. This approach aims to maximize the reliability, expressiveness, and informational value of performance assessments.
SN - 2227-7390
ER -