{
  "abstract": "Patient world models are increasingly proposed as simulators of clinical trials, and recent surveys rank their capabilities on a ladder from temporal prediction to counterfactual rollouts. Position on that ladder is currently assigned by reading papers rather than by measurement. We turn the counterfactual rung into a measured quantity. From 30,119 completed phase 2/3 trials with results posted to ClinicalTrials.gov we construct 9,996 audited within-trial contrasts from 7,207 trials, and we score a simulator on how much of its accuracy depends on knowing which intervention was tested. The instrument is an intervention-blind reference: the same model, trained on the same trials, with every intervention-identifying field withheld. Interventional skill is the error the full model removes relative to that reference. On a temporal split (878 test contrasts from 655 trials) a registry-trained gradient-boosting simulator removes a fifth of the error of a trivial baseline, but its interventional skill is -0.12 (95% CI -0.21 to -0.05): withholding the intervention makes it better, and no alternative featurisation, stratum, temporal cut or same-molecule retrieval yields an interval above zero. A frontier language model reading only the registered protocol has interventional skill 0.20 [0.08, 0.31]; smaller and open-weight models sit at the limit of detection. Perturbation tests show the large model uses the intervention as such: swapping the arms reverses the sign of its prediction in 86% of contrasts. The blind reference is itself not drug-blind (a language model recovers the withheld drug class 45% of the time from the blind prompt), so interventional skill is a conservative lower bound. A reliability layer shows that ranking by predicted error lowers it but that calibrated error control does not transfer under temporal shift. Benchmark, code, prompts and per-call model outputs are released.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2611.05907",
  "author_total": 1,
  "authors": [
    {
      "name": "Alicem Koyun"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/7basvndrfsk49.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2611.05907",
  "fulltext_url": null,
  "id": "7basvndrfsk49",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2611.05907",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "http://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.AI"
    ],
    [
      "arxiv",
      "cs.LG"
    ],
    [
      "arxiv",
      "stat.AP"
    ]
  ],
  "openalex": null,
  "pdf_url": "https://arxiv.org/pdf/2611.05907",
  "pmcid": null,
  "pmid": null,
  "published": "2026-11-04",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/7basvndrfsk49.ris",
  "sources": [
    {
      "first_seen": "2026-11-05T15:21:06Z",
      "native_id": "2611.05907",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2611.05907"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:stat.AP",
      "subfield": "statistics"
    }
  ],
  "title": "Does Your Trial Simulator Know What the Drug Does? An Intervention-Blind Test of Interventional Validity on 7,207 Registered Trials",
  "title_text": "Does Your Trial Simulator Know What the Drug Does? An Intervention-Blind Test of Interventional Validity on 7,207 Registered Trials",
  "type": "preprint",
  "updated_at": "2026-11-09T00:20:32Z",
  "url": "https://research.draconitic.com/paper/7basvndrfsk49",
  "venue": "arXiv",
  "versions": []
}