Comparing tail forecasts: the number of exceedances is the sample size
Abstract
When two forecasts of a tail, such as prediction intervals at high coverage, extreme quantiles or expected shortfall (ES), are compared by a consistent score, the score difference is known in advance except on the days on which the outcome falls beyond a forecast. The number of these exceedances, not the number of observations, is the effective sample size. For intervals at miscoverage level α, the number of observations a comparison needs grows like 1/α while the number of exceedances it needs does not, and for some pairs of forecasts no test can do better. A confidence sequence that is valid at all times and needs no distributional assumption attains this rate; Hoeffding-type procedures, calibrated by the range of the score difference, need 1/α times more. When exceedances are scarce, the Diebold-Mariano test declares the less conservative forecaster better with at least, asymptotically, the probability of observing no exceedance, whichever forecaster is actually better; the usual small-sample corrections do not remove this, while a bootstrap-t version largely does. For (value-at-risk, ES) forecasts, a less conservative ES can be shown without assumptions to be worse on the days on which it is less conservative, but never to be better without an assumption on the loss tail. In 250-day windows of 97.5% forecasts for seven stock indices, 18.4% of significant Diebold-Mariano results favouring the less conservative model contradict the ranking on the remaining days, against 6.9% of those favouring the more conservative one.
Related
- Tail reliability in probabilistic electricity price forecasts: comparing model combination and conformal prediction in the GB day-ahead market
- Comparing and measuring intermediate tail dependence
- UPPER TAIL PROBABILITY ESTIMATES OF SUPREMA OF GAUSSIAN PROCESSES BY MATRICES, A NUMBER THEORETIC EXAMPLE
- Sample size calculation comparing two survival curves in the presence of nonproportional hazards utilizing the concept of Weighted Relative Time using two different Weibull distributions
- Where should the null live? Surrogate tests for clustering of threshold-derived air-quality indices, with application to PM2.5 exceedances in Chile
BibTeX
@misc{coelho2026comparing,
title = {{Comparing tail forecasts: the number of exceedances is the sample size}},
author = {Raphael Coelho},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.40366},
eprint = {2610.40366},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.40366},
}
RIS
TY - UNPB
TI - Comparing tail forecasts: the number of exceedances is the sample size
AU - Raphael Coelho
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.40366
UR - https://arxiv.org/abs/2610.40366
AB - When two forecasts of a tail, such as prediction intervals at high coverage, extreme quantiles or expected shortfall (ES), are compared by a consistent score, the score difference is known in advance except on the days on which the outcome falls beyond a forecast. The number of these exceedances, not the number of observations, is the effective sample size. For intervals at miscoverage level α, the number of observations a comparison needs grows like 1/α while the number of exceedances it needs does not, and for some pairs of forecasts no test can do better. A confidence sequence that is valid at all times and needs no distributional assumption attains this rate; Hoeffding-type procedures, calibrated by the range of the score difference, need 1/α times more. When exceedances are scarce, the Diebold-Mariano test declares the less conservative forecaster better with at least, asymptotically, the probability of observing no exceedance, whichever forecaster is actually better; the usual small-sample corrections do not remove this, while a bootstrap-t version largely does. For (value-at-risk, ES) forecasts, a less conservative ES can be shown without assumptions to be worse on the days on which it is less conservative, but never to be better without an assumption on the loss tail. In 250-day windows of 97.5% forecasts for seven stock indices, 18.4% of significant Diebold-Mariano results favouring the less conservative model contradict the ranking on the remaining days, against 6.9% of those favouring the more conservative one.
ER -