A probability calibrator for machine-learning classifiers
Abstract
In fields such as medicine and drug discovery, the ultimate goal of a classification is not to guess a class, but to choose the optimal course of action among a set of possible ones, usually not in one-one correspondence with the set of classes. This decision-theoretic problem requires proper probabilities for the classes. Proper probabilities conditional on the features are computationally almost impossible to find in many important cases. The main idea of the present work is to calculate probabilities conditional not on the features, but on the trained classifier's output. This calculation is cheap, needs to be made only once, and provides an output-to-probability "calibrator" or "transducer" that can be applied to all future outputs of the classifier. In conjunction with problem-dependent utilities, the probabilities of the transducer allow us to find the optimal choice among the classes or among a set of more general decisions, by means of expected-utility maximization. This idea is demonstrated in a simplified drug-discovery problem with a highly imbalanced dataset. The transducer and utility maximization together always lead to improved results, sometimes close to the theoretical maximum, for all sets of problem-dependent utilities. The one-time-only calculation of the transducer also provides, automatically: (i) a quantification of the uncertainty about the transducer itself; (ii) the expected utility of the augmented algorithm (including its uncertainty), which can be used for algorithm selection; (iii) the possibility of using the algorithm in a ’generative mode’, useful if the training dataset is biased.
Related
- Reproducible benchmarking of four machine-learning classifiers on a large synthetic heart-disease competition dataset
- Calibrating Machine Learning Approaches for Probability Estimation Without Calibration Data.
- Constructive Probability, Foundations and Numerical Analysis for mathematical finance and machine learning
- Ensemble-based support vector machine classifiers as an efficient tool for quality assessment of beef fillets from electronic nose data
- A Unified Framework for Training-Dataset Aggregation and Exact Machine Unlearning via Gibbs Probability Measures
BibTeX
@misc{dyrland2026probability,
title = {{A probability calibrator for machine-learning classifiers}},
author = {Kjetil Dyrland and Alexander Selvikvåg Lundervold and PierGianLuca Porta Mana},
howpublished = {OSF Preprints},
year = {2026},
month = oct,
doi = {10.31219/osf.io/vct9y_v2},
url = {https://osf.io/vct9y_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A probability calibrator for machine-learning classifiers
AU - Kjetil Dyrland
AU - Alexander Selvikvåg Lundervold
AU - PierGianLuca Porta Mana
PY - 2026
DA - 2026/10/18
PB - OSF Preprints
DO - 10.31219/osf.io/vct9y_v2
UR - https://osf.io/vct9y_v2/
AB - In fields such as medicine and drug discovery, the ultimate goal of a classification is not to guess a class, but to choose the optimal course of action among a set of possible ones, usually not in one-one correspondence with the set of classes. This decision-theoretic problem requires proper probabilities for the classes. Proper probabilities conditional on the features are computationally almost impossible to find in many important cases. The main idea of the present work is to calculate probabilities conditional not on the features, but on the trained classifier's output. This calculation is cheap, needs to be made only once, and provides an output-to-probability "calibrator" or "transducer" that can be applied to all future outputs of the classifier. In conjunction with problem-dependent utilities, the probabilities of the transducer allow us to find the optimal choice among the classes or among a set of more general decisions, by means of expected-utility maximization. This idea is demonstrated in a simplified drug-discovery problem with a highly imbalanced dataset. The transducer and utility maximization together always lead to improved results, sometimes close to the theoretical maximum, for all sets of problem-dependent utilities. The one-time-only calculation of the transducer also provides, automatically: (i) a quantification of the uncertainty about the transducer itself; (ii) the expected utility of the augmented algorithm (including its uncertainty), which can be used for algorithm selection; (iii) the possibility of using the algorithm in a ’generative mode’, useful if the training dataset is biased.
ER -