A Multiverse of Good and Bad Controls: Candidate Causal Graphs for Interpreting Model Robustness Analysis
Abstract
Model robustness analysis estimates an effect across a multiverse of specifications that pools control sets identifying the declared estimand with sets that condition on mediators or colliders. We propose stating rival assumptions about contested controls as a small set of candidate causal graphs, enumerating the adjustment sets each graph licenses, and reporting robustness metrics conditional on each graph. A finite-mixture identity splits the licensed multiverse's dispersion into within-graph and between-graph components; the between-graph share is a conditional descriptive summary whose reading depends on the candidate set, the weights, and a common estimand. Simulations examine misleading pooled robustness assessments and the limits of the decomposition. Applications to hurricane fatalities, job training, and union wages show fragility that survives every graph, instability produced by unlicensed specifications, and a fragility verdict concealing a significant premium in each adjustment-identified candidate world. An R package implements the workflow.
Related
- Minimum Specification Perturbation: Robustness as Distance-to-Falsification in Causal Inference
- Causal Robustness from Synthetic Environments
- Metatheoretical multiverse analysis: Improving the reliability of research with theories-as-data
- Instability in Patient Clustering: A Multiverse Analysis of Unsupervised Clustering in the CENTER-TBI cohort
- Interpreting relative utility for probabilistic predictions
BibTeX
@misc{okubo2026multiverse,
title = {{A Multiverse of Good and Bad Controls: Candidate Causal Graphs for Interpreting Model Robustness Analysis}},
author = {Shoki Okubo},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.16618},
eprint = {2609.16618},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.16618},
}
RIS
TY - UNPB
TI - A Multiverse of Good and Bad Controls: Candidate Causal Graphs for Interpreting Model Robustness Analysis
AU - Shoki Okubo
PY - 2026
DA - 2026/09/15
PB - arXiv
DO - 10.48550/arxiv.2609.16618
UR - https://arxiv.org/abs/2609.16618
AB - Model robustness analysis estimates an effect across a multiverse of specifications that pools control sets identifying the declared estimand with sets that condition on mediators or colliders. We propose stating rival assumptions about contested controls as a small set of candidate causal graphs, enumerating the adjustment sets each graph licenses, and reporting robustness metrics conditional on each graph. A finite-mixture identity splits the licensed multiverse's dispersion into within-graph and between-graph components; the between-graph share is a conditional descriptive summary whose reading depends on the candidate set, the weights, and a common estimand. Simulations examine misleading pooled robustness assessments and the limits of the decomposition. Applications to hurricane fatalities, job training, and union wages show fragility that survives every graph, instability produced by unlicensed specifications, and a fragility verdict concealing a significant premium in each adjustment-identified candidate world. An R package implements the workflow.
ER -