Grouped Validation and Conformal Calibration for Clustered Cross-National Data with an Application to Harmonized Learning Outcomes
- 1Department of Mathematics, College of Science, Taibah University, Madinah 42353, Saudi Arabia
Abstract
Clustered cross-national panels pose two problems: observation-level validation can be optimistic when the same economy appears in training and evaluation, while standard split conformal coverage requires exchangeability that pooled cells from heterogeneous economies do not justify. We study prediction, attribution, and interval calibration using 1565 Harmonized Learning Outcomes results from 144 economies. Performance is estimated by nested cross-validation grouped by economy and a temporal split; explanations are computed on evaluation folds using Shapley values and permutation importance. XGBoost attains R2=0.703, versus 0.593 and 0.595 for linear baselines, and R2=0.819 temporally. Under a country-held-out conformal design, three calibration rules are compared on identical predictions. At 90% nominal coverage, pooled calibration yields a cell-weighted and country-weighted coverage of 0.872 and 0.794 with a mean width of 118.9; equal-economy-calibration yields of 0.930 and 0.875 at 148.5; and subsampling yields of 0.961 and 0.937 at 180.4. Across 50 repeated calibration splits with re-selected XGBoost hyperparameters, the width ordering of the rules holds in every replicate, the cell-weighted coverage gain from pooled to equal-economy calibration is positive in 49, and the width increase averages 22 points; in the primary analysis, selecting the model class inside each development partition returns the same models in every fold. The observed coverage for lower-middle-income economies is below nominal under the first two rules and persistently so across calibration splits, although the economy-bootstrap interval under equal-economy calibration does not exclude the nominal level. Attribution, conditional on the included indicators and fitted models, emphasizes mortality under age five and adult schooling. These results show that cluster-aware validation and both coverage functionals materially affect predictive reliability.
Related
- Estimating marginal association between paired outcomes in clustered data with an application to analyzing caries and periodontal disease data from the National Health and Nutrition Examination Survey
- When Does Trimming Help Conformal Prediction? A Retained-Law Diagnostic under Calibration Contamination
- Conformal Calibration Transfer
- Multi-Variable Conformal Prediction: Optimizing Prediction Sets using Multiple Calibration Variables
- Announced Breaks Separate Conformal Reliability from Frequency Calibration
BibTeX
@article{allohibi2026grouped,
title = {{Grouped Validation and Conformal Calibration for Clustered Cross-National Data with an Application to Harmonized Learning Outcomes}},
author = {Jeza Allohibi},
journal = {Mathematics},
year = {2026},
month = nov,
doi = {10.3390/math14214055},
url = {https://www.mdpi.com/2227-7390/14/21/4055},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Grouped Validation and Conformal Calibration for Clustered Cross-National Data with an Application to Harmonized Learning Outcomes
AU - Jeza Allohibi
PY - 2026
DA - 2026/11/05
JO - Mathematics
DO - 10.3390/math14214055
UR - https://www.mdpi.com/2227-7390/14/21/4055
AB - Clustered cross-national panels pose two problems: observation-level validation can be optimistic when the same economy appears in training and evaluation, while standard split conformal coverage requires exchangeability that pooled cells from heterogeneous economies do not justify. We study prediction, attribution, and interval calibration using 1565 Harmonized Learning Outcomes results from 144 economies. Performance is estimated by nested cross-validation grouped by economy and a temporal split; explanations are computed on evaluation folds using Shapley values and permutation importance. XGBoost attains R2=0.703, versus 0.593 and 0.595 for linear baselines, and R2=0.819 temporally. Under a country-held-out conformal design, three calibration rules are compared on identical predictions. At 90% nominal coverage, pooled calibration yields a cell-weighted and country-weighted coverage of 0.872 and 0.794 with a mean width of 118.9; equal-economy-calibration yields of 0.930 and 0.875 at 148.5; and subsampling yields of 0.961 and 0.937 at 180.4. Across 50 repeated calibration splits with re-selected XGBoost hyperparameters, the width ordering of the rules holds in every replicate, the cell-weighted coverage gain from pooled to equal-economy calibration is positive in 49, and the width increase averages 22 points; in the primary analysis, selecting the model class inside each development partition returns the same models in every fold. The observed coverage for lower-middle-income economies is below nominal under the first two rules and persistently so across calibration splits, although the economy-bootstrap interval under equal-economy calibration does not exclude the nominal level. Attribution, conditional on the included indicators and fitted models, emphasizes mortality under age five and adult schooling. These results show that cluster-aware validation and both coverage functionals materially affect predictive reliability.
SN - 2227-7390
ER -