Calibration scoring under dependent seed variables in Cooke's Classical Model: a Mehler-Hermite design-effect correction
Abstract
Cooke's Classical Model scores the statistical accuracy of experts by the p-value of a likelihood ratio test that compares the bin frequencies of realized seed variables with nominal probabilities, using a χ2 reference. This calibration score drives performance-based weights and implicitly assumes independent seed variables, although seeds often share a common driver, such as the same indicator in several regions or periods. Under dependence the statistic converges to a weighted sum of χ21 variables, and positive dependence makes the classical score anti-conservative. Under a Gaussian-copula working model, a Mehler-Hermite expansion gives the long-run covariance of the bin indicators in closed form, defines a design effect and an effective number of seed variables, and shows that the first-order correction coincides with the ad hoc calibration power of existing software. Rao-Scott-type and parametric-bootstrap corrected scores are proposed, together with a restricted maximum likelihood estimator of the nuisance dependence, computed from rank normal scores and invariant to location-scale miscalibration, and a prepivoted bootstrap that propagates its uncertainty. In simulations the classical score rejects up to 68\% of calibrated experts at a nominal 5\% level, whereas the corrected scores remove most of the excess rejection. In 45 published studies, design effects of 1.25-2 shift more than 5\% of the optimized weight in 29-44\% of studies. In three social-science studies with structured seeds, the baseline estimates imply that 18-21 seed variables carry the information of only 11-13 independent ones, although conclusions about individual experts depend on how the dependence is specified.
Related
- Multi-Variable Conformal Prediction: Optimizing Prediction Sets using Multiple Calibration Variables
- Teacher Supervision, Fixed Representations and Calibration: Exact Risk Calculations for Candidate-Scoring Decision Models
- Stein's functional method for weakly dependent random variables
- Sums of two dependent uniform random variables: a characterization by joint tail bounds
- Rerandomization and high-dimensional adjustment with assignment-dependent auxiliary variables under network interference
BibTeX
@misc{calleo2026calibration,
title = {{Calibration scoring under dependent seed variables in Cooke's Classical Model: a Mehler-Hermite design-effect correction}},
author = {Yuri Calleo},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.28831},
eprint = {2610.28831},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28831},
}
RIS
TY - UNPB
TI - Calibration scoring under dependent seed variables in Cooke's Classical Model: a Mehler-Hermite design-effect correction
AU - Yuri Calleo
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.28831
UR - https://arxiv.org/abs/2610.28831
AB - Cooke's Classical Model scores the statistical accuracy of experts by the p-value of a likelihood ratio test that compares the bin frequencies of realized seed variables with nominal probabilities, using a χ^2 reference. This calibration score drives performance-based weights and implicitly assumes independent seed variables, although seeds often share a common driver, such as the same indicator in several regions or periods. Under dependence the statistic converges to a weighted sum of χ^2_1 variables, and positive dependence makes the classical score anti-conservative. Under a Gaussian-copula working model, a Mehler-Hermite expansion gives the long-run covariance of the bin indicators in closed form, defines a design effect and an effective number of seed variables, and shows that the first-order correction coincides with the ad hoc calibration power of existing software. Rao-Scott-type and parametric-bootstrap corrected scores are proposed, together with a restricted maximum likelihood estimator of the nuisance dependence, computed from rank normal scores and invariant to location-scale miscalibration, and a prepivoted bootstrap that propagates its uncertainty. In simulations the classical score rejects up to 68\% of calibrated experts at a nominal 5\% level, whereas the corrected scores remove most of the excess rejection. In 45 published studies, design effects of 1.25-2 shift more than 5\% of the optimized weight in 29-44\% of studies. In three social-science studies with structured seeds, the baseline estimates imply that 18-21 seed variables carry the information of only 11-13 independent ones, although conclusions about individual experts depend on how the dependence is specified.
ER -