Null-Corrected Conditional Coverage in Materials Property Prediction: A Five-Dataset Replication
Abstract
Uncertainty quantification for materials property prediction increasingly reports conditional coverage, coverage within strata of the predicted value, alongside marginal coverage. The worst-bin statistics used for this are downward-biased in finite samples, a known defect that applied materials work does not generally correct for. We replicate a single-dataset conditional coverage study across five regression datasets (n = 437 to 21,263), seven interval methods and 50 random splits each, correcting the worst-bin statistic against its expected value under exact conditional validity. The null is simulated from realized per-bin test counts and referenced to each method's own marginal coverage, separating marginal miscalibration from uneven coverage. On the smallest dataset the correction runs from 0.061 to 0.096, between a third and two thirds of the shortfalls actually reported, against 0.009 to 0.011 on the two largest. Split conformal prediction's failure survives the correction on four of five datasets; two conclusions of the antecedent study do not. Conformalized quantile regression, characterized there as a negative case, fails on one dataset only. Cross-fitted normalized conformal prediction, grouped there with the successful methods, fails on three of five. Group-conditional (Mondrian) conformal prediction stays below our significance criterion on every dataset, yet exceeds the null on three of five under an octile refinement of the variable it calibrates on. A level-matched ablation locates that uniformity in group-conditional calibration rather than in the elevated quantile level, and removes Mondrian's apparent width cost entirely. Conditional coverage evaluations should report the null distribution of the worst-bin statistic used, the method's realized marginal coverage, and the resolution of the evaluation partition relative to any partition the method calibrates on.
Related
- Conformal Prediction and Conditional Coverage for Tabular Foundation Models
- How Synthetic Labels Improve Conformal Prediction: A Perspective on Conditional Coverage
- Sharp training-conditional coverage for conformal prediction under covariate shift
- Conformal Prediction for Multi-Step Forecasting: Conditional Coverage and Sharp Score Calibration Rates
- Bias-Corrected Estimators for Joint Entropy, Conditional Entropy, and Mutual Information in the Discrete Bivariate Case: Theory and an Application to Motor Insurance
BibTeX
@misc{boruah2026nullcorrected,
title = {{Null-Corrected Conditional Coverage in Materials Property Prediction: A Five-Dataset Replication}},
author = {Irene Boruah},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.14521},
eprint = {2611.14521},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.14521},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Null-Corrected Conditional Coverage in Materials Property Prediction: A Five-Dataset Replication
AU - Irene Boruah
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.14521
UR - https://arxiv.org/abs/2611.14521
AB - Uncertainty quantification for materials property prediction increasingly reports conditional coverage, coverage within strata of the predicted value, alongside marginal coverage. The worst-bin statistics used for this are downward-biased in finite samples, a known defect that applied materials work does not generally correct for. We replicate a single-dataset conditional coverage study across five regression datasets (n = 437 to 21,263), seven interval methods and 50 random splits each, correcting the worst-bin statistic against its expected value under exact conditional validity. The null is simulated from realized per-bin test counts and referenced to each method's own marginal coverage, separating marginal miscalibration from uneven coverage. On the smallest dataset the correction runs from 0.061 to 0.096, between a third and two thirds of the shortfalls actually reported, against 0.009 to 0.011 on the two largest. Split conformal prediction's failure survives the correction on four of five datasets; two conclusions of the antecedent study do not. Conformalized quantile regression, characterized there as a negative case, fails on one dataset only. Cross-fitted normalized conformal prediction, grouped there with the successful methods, fails on three of five. Group-conditional (Mondrian) conformal prediction stays below our significance criterion on every dataset, yet exceeds the null on three of five under an octile refinement of the variable it calibrates on. A level-matched ablation locates that uniformity in group-conditional calibration rather than in the elevated quantile level, and removes Mondrian's apparent width cost entirely. Conditional coverage evaluations should report the null distribution of the worst-bin statistic used, the method's realized marginal coverage, and the resolution of the evaluation partition relative to any partition the method calibrates on.
ER -