Where Causal Discovery Breaks on Real Data: Measuring the Assumptions One Property at a Time
Abstract
Causal discovery is usually evaluated on simulators or on systems whose ground truth follows from physics or design. When it fails on other real data, the failure is attributed to assumptions that may not hold, but which ones fail, and by how much, is rarely measured. We measure the observable properties of the data they bear on, one property at a time, on a real panel. The case study follows 2,901 US public firms over 31 quarters, with 71 types of corporate action from SEC 8-K filings, 11 financial-statement items, and an expert graph over 651 action-state pairs. The panel breaks measurable requirements of discovery: tails that survive winsorisation, balance-sheet items and revenue close to random walks, and interventions too rare to detect effects of a tenth of a standard deviation in 40 of 71 actions. Even so, a marginal test ranks expert-labelled direct above not-direct pairs within each action (AUROC 0.658) better than every null we built, so the panel does carry information about which pairs are direct. Tuned discovery methods, in configurations feasible at this panel size, recover less of it than a simple pairwise cross-correlation when pairs are ranked within each action, and no more when they are ranked within each target, which removes what the target alone predicts. Most direct pairs (132 of 222) involve actions below the support needed for 80% power against a 0.1-SD effect. For flow targets, conditioning on the target's other expert-labelled parents, often filed alongside the action, erodes the marginal discrimination, and most not-direct associations survive it: even with the graph's parent sets supplied, the expected separation does not emerge. For each property, we give a diagnostic computable before any discovery method is run.
Related
- A General Approach for Assessing Sensitivity to Violations of Causal Assumptions
- A Subjective Bayesian Approach to Address Unchecked Assumptions in a Causal Analysis
- Unit-level aggregation breaks coverage in causal foundation models
- Measuring Gift Card Program Incrementality via Causal Data Fusion
- Interpretable Causal Discovery via Causal-Effect Constraints
BibTeX
@misc{jonnalagedda2026where,
title = {{Where Causal Discovery Breaks on Real Data: Measuring the Assumptions One Property at a Time}},
author = {Padmaja Jonnalagedda and Avinash Baidya and Ruocheng Guo and Kamalika Das},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.00484},
eprint = {2611.00484},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.00484},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Where Causal Discovery Breaks on Real Data: Measuring the Assumptions One Property at a Time
AU - Padmaja Jonnalagedda
AU - Avinash Baidya
AU - Ruocheng Guo
AU - Kamalika Das
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.00484
UR - https://arxiv.org/abs/2611.00484
AB - Causal discovery is usually evaluated on simulators or on systems whose ground truth follows from physics or design. When it fails on other real data, the failure is attributed to assumptions that may not hold, but which ones fail, and by how much, is rarely measured. We measure the observable properties of the data they bear on, one property at a time, on a real panel. The case study follows 2,901 US public firms over 31 quarters, with 71 types of corporate action from SEC 8-K filings, 11 financial-statement items, and an expert graph over 651 action-state pairs. The panel breaks measurable requirements of discovery: tails that survive winsorisation, balance-sheet items and revenue close to random walks, and interventions too rare to detect effects of a tenth of a standard deviation in 40 of 71 actions. Even so, a marginal test ranks expert-labelled direct above not-direct pairs within each action (AUROC 0.658) better than every null we built, so the panel does carry information about which pairs are direct. Tuned discovery methods, in configurations feasible at this panel size, recover less of it than a simple pairwise cross-correlation when pairs are ranked within each action, and no more when they are ranked within each target, which removes what the target alone predicts. Most direct pairs (132 of 222) involve actions below the support needed for 80% power against a 0.1-SD effect. For flow targets, conditioning on the target's other expert-labelled parents, often filed alongside the action, erodes the marginal discrimination, and most not-direct associations survive it: even with the graph's parent sets supplied, the expected separation does not emerge. For each property, we give a diagnostic computable before any discovery method is run.
ER -