On the Calibration of Crowdsourcing Aggregators
Abstract
Calibration has been extensively studied for classifiers, but remains largely unexplored for crowdsourcing aggregation methods, which are increasingly used for making probabilistic decisions. To the best of our knowledge, we provide the first theoretical and empirical study of calibration in label aggregation. To our great surprise, we find that widely used aggregation methods can be severely miscalibrated, and in opposite directions: Majority Vote is underconfident while model-based aggregators are overconfident. We prove that the Bayesian aggregator is perfectly calibrated under its data-generating model. We also show that exact calibration of weighted majority votes requires highly restrictive conditions, which helps explain why Majority Vote is generally miscalibrated in realistic crowds. Empirically, we study calibration across $39$ human crowdsourcing datasets and LLM-based crowds. In both cases, we observe systematic miscalibration for aggregation models such as Majority Vote, Dawid--Skene, GLAD, and CrowdFM. Finally, we propose a post-hoc calibration procedure that \emph{aggregates the aggregators}: for any aggregation function ϕ, we introduce a transformation that first combines its predictive distribution with those of complementary aggregators, before applying temperature scaling. This simple and interpretable transformation makes temperature scaling more effective than when applied directly to ϕ. It improves calibration, especially in the high-redundancy regime where there are many annotations per item, while generally preserving or even improving accuracy.
Related
- Pathwise Shrinkage and Probabilistic Calibration of the Multiplicative Half-Cauchy Process
- The quantitative isoperimetric inequality: A calibration argument
- Does Calibration Pick the Right Model? Likelihood Evidence from Income Risk
- A Three-Way Testing Framework for Quantifying Epistemic Calibration Uncertainty in SBI
- On the Calibration of Isotonic Distributional Regression
BibTeX
@misc{singer2026calibration,
title = {{On the Calibration of Crowdsourcing Aggregators}},
author = {Gabriel Singer and Reda Arab and Maïssa Frikha and Samuel Gruffaz and Olivier Vo Van and Nicolas Vayatis and Argyris Kalogeratos},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.16815},
eprint = {2610.16815},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.16815},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - On the Calibration of Crowdsourcing Aggregators
AU - Gabriel Singer
AU - Reda Arab
AU - Maïssa Frikha
AU - Samuel Gruffaz
AU - Olivier Vo Van
AU - Nicolas Vayatis
AU - Argyris Kalogeratos
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2610.16815
UR - https://arxiv.org/abs/2610.16815
AB - Calibration has been extensively studied for classifiers, but remains largely unexplored for crowdsourcing aggregation methods, which are increasingly used for making probabilistic decisions. To the best of our knowledge, we provide the first theoretical and empirical study of calibration in label aggregation. To our great surprise, we find that widely used aggregation methods can be severely miscalibrated, and in opposite directions: Majority Vote is underconfident while model-based aggregators are overconfident. We prove that the Bayesian aggregator is perfectly calibrated under its data-generating model. We also show that exact calibration of weighted majority votes requires highly restrictive conditions, which helps explain why Majority Vote is generally miscalibrated in realistic crowds. Empirically, we study calibration across $39$ human crowdsourcing datasets and LLM-based crowds. In both cases, we observe systematic miscalibration for aggregation models such as Majority Vote, Dawid--Skene, GLAD, and CrowdFM. Finally, we propose a post-hoc calibration procedure that \emph{aggregates the aggregators}: for any aggregation function ϕ, we introduce a transformation that first combines its predictive distribution with those of complementary aggregators, before applying temperature scaling. This simple and interpretable transformation makes temperature scaling more effective than when applied directly to ϕ. It improves calibration, especially in the high-redundancy regime where there are many annotations per item, while generally preserving or even improving accuracy.
ER -