Normalized risk-based evaluation of machine learning–based classification models: A multiclass approach with applications in medical devices
- 1Faculty I, Computer Science & Applications, Furtwangen University, Germany
- 2Faculty II, Engineering & Technology, Furtwangen University, Germany
Abstract
Objective: Machine learning is increasingly integrated into safety-critical domains such as medical applications. In this context, regulatory frameworks require the assessment and minimization of risks associated with incorrect model predictions. However, classical evaluation methods often focus on quantifying error frequencies, which do not reflect the heterogeneous impact of different types of errors. To address this deficiency, we elaborate an approach for assessing machine learning–based multiclass classification models that conforms to regulatory requirements in the field of medical devices. Additionally, we aim to demonstrate its applicability in a concrete application scenario as an exemplary reference.
Methods: Herein, we provide weighted balanced accuracy WBAn as a risk-based approach for assessing machine learning models for multiclass classification. WBAn is systematically based on regulatory requirements for medical devices. It incorporates core properties such as the representation of risks as a multiplicative combination of likelihood and severity and the relationship between development and real-world scenarios. We demonstrate the application of WBAn using X-ray based detection of lung diseases as a reference scenario.
Results: Our analysis shows that WBAn achieves a risk-based assessment in the lung disease scenario as a reference. WBAn systematically operationalizes the definition of risk from corresponding regulatory requirements and allows the assessment to be adapted according to risk levels that have to be elaborated in the context of the risk-management process.
Conclusion: Although this study uses only a single use case to demonstrate the practical applicability of our approach, it shows the impact of a risk-based approach when assessing the performance of machine learning multiclass classification models in medical applications. Without such an approach, regulatory requirements cannot be implemented in a fully comprehensive way. Consequently, the clinical impact cannot be adequately assessed when applying the model in real-world scenarios.
Related
- Machine learning-based multiclass classification of cognitive decline using ACE-III assessment data: a comparative study with SHAP explainability
- DualXBrNet: A Hybrid Dual-Path Cross-Level Fusion Framework for Multiclass Brain Tumor Classification Using MRI Images
- COVLIAS 3.5: integration of attention-based segmentation technique with fuzzy dilated convolutional neural networks for improved classification of chest X-ray scans for multiclass pneumonia diagnosis
- Comparative Evaluation of Multiple Deep Learning Models for Six-Class Keratitis Classification
- Deep Learning-Based Classification of Melanoma Using Dermoscopic Images: Comparative Evaluation of Convolutional Neural Network and DenseNet Models.
BibTeX
@article{wiggerthale2026normalized,
title = {{Normalized risk-based evaluation of machine learning–based classification models: A multiclass approach with applications in medical devices}},
author = {Julius Wiggerthale and Martin Haimerl and Christoph Reich},
journal = {The Journal of International Medical Research},
year = {2026},
month = sep,
doi = {10.1177/03000605261486667},
url = {https://doi.org/10.1177/03000605261486667},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Normalized risk-based evaluation of machine learning–based classification models: A multiclass approach with applications in medical devices
AU - Julius Wiggerthale
AU - Martin Haimerl
AU - Christoph Reich
PY - 2026
DA - 2026/09/28
JO - The Journal of International Medical Research
DO - 10.1177/03000605261486667
UR - https://doi.org/10.1177/03000605261486667
AB - Objective: Machine learning is increasingly integrated into safety-critical domains such as medical applications. In this context, regulatory frameworks require the assessment and minimization of risks associated with incorrect model predictions. However, classical evaluation methods often focus on quantifying error frequencies, which do not reflect the heterogeneous impact of different types of errors. To address this deficiency, we elaborate an approach for assessing machine learning–based multiclass classification models that conforms to regulatory requirements in the field of medical devices. Additionally, we aim to demonstrate its applicability in a concrete application scenario as an exemplary reference. Methods: Herein, we provide weighted balanced accuracy WBAn as a risk-based approach for assessing machine learning models for multiclass classification. WBAn is systematically based on regulatory requirements for medical devices. It incorporates core properties such as the representation of risks as a multiplicative combination of likelihood and severity and the relationship between development and real-world scenarios. We demonstrate the application of WBAn using X-ray based detection of lung diseases as a reference scenario. Results: Our analysis shows that WBAn achieves a risk-based assessment in the lung disease scenario as a reference. WBAn systematically operationalizes the definition of risk from corresponding regulatory requirements and allows the assessment to be adapted according to risk levels that have to be elaborated in the context of the risk-management process. Conclusion: Although this study uses only a single use case to demonstrate the practical applicability of our approach, it shows the impact of a risk-based approach when assessing the performance of machine learning multiclass classification models in medical applications. Without such an approach, regulatory requirements cannot be implemented in a fully comprehensive way. Consequently, the clinical impact cannot be adequately assessed when applying the model in real-world scenarios.
SN - 0300-0605
SN - 1473-2300
ER -