When bad adjustment looks good: what goes wrong in Plasmode 0.1.0 simulations
Abstract
Comparisons of confounding-control methods are informative only when simulated data contain the treatment-covariate relationship those methods address. We assessed whether Plasmode 0.1.0 preserves that relationship, uses the returned exposure to generate outcomes, and reports the effect implied by its generating model. We inspected the source and ran the generator under identical seeds with source rows as supplied, sorted by observed exposure, or using an alignment correction. We ran 200 null-effect replicates per strategy in a synthetic cohort and the SUPPORT/Right Heart Catheterisation cohort. A sampled subject usually received another subject's exposure probability. In the synthetic cohort, source-model AUC was 0.503 before and 0.841 after alignment. Crude risk-difference bias moved from +0.0051 to +0.3037. Other orders preserved or reversed the association. After alignment, deliberately misspecified estimators retained risk-difference biases of 0.10-0.11. Correctly specified estimators were nearly unbiased. With the unmodified generator, all five appeared similarly unbiased. At an injected odds ratio of 2, the conditional log odds ratio appeared on the observed exposure (+0.696; target 0.693), not the returned exposure (+0.017). Misalignment can weaken, preserve, or reverse designed confounding and thereby distort method comparisons. Alignment correction does not repair outcome-exposure decoupling, disagreement between the reported and generating effects, or marginal miscalibration.
Related
- When Does Inexact Matching Ensure Balance and Inference without Adjustment?
- A Plasmode Benchmark Can Refute but Cannot Certify: Projection and Regularity in Causal-Inference Simulation
- Estimand Alignment and Implementation Sensitivity in Causal Mediation Analysis of Environmental Mixtures: Simulation, Plasmode, and Applied Study.
- Adaptive Patching Is Harder Than It Looks For Time-Series Forecasting
- Lane-Emden equation for the superposition of a local and nonlocal operator with the "wrong sign"
BibTeX
@misc{karim2026when,
title = {{When bad adjustment looks good: what goes wrong in Plasmode 0.1.0 simulations}},
author = {M. Ehsan Karim},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.24053},
eprint = {2609.24053},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.24053},
}
RIS
TY - UNPB
TI - When bad adjustment looks good: what goes wrong in Plasmode 0.1.0 simulations
AU - M. Ehsan Karim
PY - 2026
DA - 2026/09/21
PB - arXiv
DO - 10.48550/arxiv.2609.24053
UR - https://arxiv.org/abs/2609.24053
AB - Comparisons of confounding-control methods are informative only when simulated data contain the treatment-covariate relationship those methods address. We assessed whether Plasmode 0.1.0 preserves that relationship, uses the returned exposure to generate outcomes, and reports the effect implied by its generating model. We inspected the source and ran the generator under identical seeds with source rows as supplied, sorted by observed exposure, or using an alignment correction. We ran 200 null-effect replicates per strategy in a synthetic cohort and the SUPPORT/Right Heart Catheterisation cohort. A sampled subject usually received another subject's exposure probability. In the synthetic cohort, source-model AUC was 0.503 before and 0.841 after alignment. Crude risk-difference bias moved from +0.0051 to +0.3037. Other orders preserved or reversed the association. After alignment, deliberately misspecified estimators retained risk-difference biases of 0.10-0.11. Correctly specified estimators were nearly unbiased. With the unmodified generator, all five appeared similarly unbiased. At an injected odds ratio of 2, the conditional log odds ratio appeared on the observed exposure (+0.696; target 0.693), not the returned exposure (+0.017). Misalignment can weaken, preserve, or reverse designed confounding and thereby distort method comparisons. Alignment correction does not repair outcome-exposure decoupling, disagreement between the reported and generating effects, or marginal miscalibration.
ER -