Calibration, accuracy, and domain variation across one million collective forecasts from prediction platforms
- 1Private address, Indianapolis, IN
- 2The School of Psychological Science, Tel Aviv University, Tel Aviv 6997801, Israel
Abstract
Using approximately one million resolved contracts and questions across four collective forecasting platforms, twelve domains, and time horizons ranging from hours to years, we evaluate the calibration and accuracy of collective forecasts in real-world settings. We found that forecast probabilities were overall highly calibrated: events assigned a probability of p% occurred approximately p% of the time. This broad pattern was evident across platforms and many domains, although fine-grained analyses revealed systematic tendencies toward over- or underestimation in highly consequential areas. While this does not mean that forecasting platforms provide certainty about individual events, our findings suggest that large-scale aggregation of judgments contains meaningful information about real-world uncertainty; when examined in aggregate, the predictions generated by the platforms provided informative estimates of event likelihood.
Related
- Benchmarking cell segmentation in two Spatialomics platforms with open-source software
- Beyond correlation: Intraclass correlation and bland-altman analysis reveal poor absolute agreement across multiplex cytokine immunoassay platforms
- Hybrid exosome-liposome nanovesicles as delivery platforms for therapeutic applications: merging the best of both worlds.
- Estimating the threat of AI-agent responding across online survey platforms.
- Biological Parts in Yeast Synthetic Biology: From Regulatory Elements to Predictive Design Platforms
BibTeX
@article{dickerson2026calibration,
title = {{Calibration, accuracy, and domain variation across one million collective forecasts from prediction platforms}},
author = {Justin Dickerson and Hilla Shinitzky and Maayan Peer and Mattan S. Ben-Shachar and Michael Gilead},
journal = {Proceedings of the National Academy of Sciences of the United States of America},
year = {2026},
month = oct,
doi = {10.1073/pnas.2615627123},
url = {https://doi.org/10.1073/pnas.2615627123},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - Calibration, accuracy, and domain variation across one million collective forecasts from prediction platforms
AU - Justin Dickerson
AU - Hilla Shinitzky
AU - Maayan Peer
AU - Mattan S. Ben-Shachar
AU - Michael Gilead
PY - 2026
DA - 2026/10/19
JO - Proceedings of the National Academy of Sciences of the United States of America
DO - 10.1073/pnas.2615627123
UR - https://doi.org/10.1073/pnas.2615627123
AB - Using approximately one million resolved contracts and questions across four collective forecasting platforms, twelve domains, and time horizons ranging from hours to years, we evaluate the calibration and accuracy of collective forecasts in real-world settings. We found that forecast probabilities were overall highly calibrated: events assigned a probability of p% occurred approximately p% of the time. This broad pattern was evident across platforms and many domains, although fine-grained analyses revealed systematic tendencies toward over- or underestimation in highly consequential areas. While this does not mean that forecasting platforms provide certainty about individual events, our findings suggest that large-scale aggregation of judgments contains meaningful information about real-world uncertainty; when examined in aggregate, the predictions generated by the platforms provided informative estimates of event likelihood.
SN - 0027-8424
SN - 1091-6490
ER -