Cross-Domain Chest Radiograph Report Generation Under Dataset Shift: Bidirectional Audit Study.
- 1School of Electronics and Information Engineering, Sichuan University, Sichuan University Wangjiang Campus, 24 South Section 1, First Ring Road, Chengdu, Sichuan, 610065, China, 86 1-808-110-0530.
- 2School of Communication and Information Engineering, Shanghai University, Shanghai, China.
Abstract
Background
Cross-domain chest radiograph report generation may produce formally valid reports that are nevertheless repetitive, weakly case-specific, or missing clinically relevant findings. These failures can be missed when evaluation relies on a single language-overlap or clinical-label score.Objective
This study aimed to evaluate cross-domain report generation with a bidirectional layered audit, characterize direction-specific and end point-specific failure profiles under a composite training-side control and 3 decoding policies, and explore whether selected audit dimensions remained informative for a contemporary medical vision language model.Methods
A fixed R2Gen-style system was evaluated bidirectionally on 575 IU X-ray and 401 MIMIC-CXR (Medical Information Mart for Intensive Care Chest X-Ray) target records. The audit included surface validity, complete-report recurrence and concentration, a predefined prefix end point, lexical diversity, official Stanford CheXpert/NegBio labels, paired cluster-bootstrap inference, and blinded review by 2 medical trainees. The primary human audit included 210 reports from 70 randomly sampled cases, with blinded adjudication of unresolved primary end point disagreements. A 60-report targeted panel was used for failure-mode analysis. Separately, a fixed MedGemma 1.5 4B configuration generated reports for all 976 records.Results
In MIMIC-CXR-to-IU X-ray transfer, out of 575 reports, ON (protocol-defined training-side control enabled)+B4 (beam search with a beam size of 4) reduced the predefined recurrent prefix from 575 (100%) to 8 (1.4%; Holm-adjusted P=.002). However, 569 (99.0%) ON+B4 outputs still belonged to exact duplicate groups. In IU X-ray-to-MIMIC-CXR transfer, the 14-category CheXpert/NegBio micro-F₁ was higher for ON+B4 than for OFF (protocol-defined training-side control disabled)+B4 (0.1687 vs 0.0744; Holm-adjusted P=.01). This apparent advantage was not broad-based: after cardiomegaly was excluded in a post hoc descriptive sensitivity analysis, the difference changed from +0.0944 to -0.0610. Text-level tracing showed that the original contrast was mainly driven by a labeler response to recurrent, semantically normal heart-size wording. Across the primary human-audit cells, major omissions occurred in 12 to 15 out of 35 reports, and major unsupported findings occurred in 10 to 16 of 35 reports, with no consistent configuration ordering observed. The exploratory MedGemma micro-F₁ was 0.4722 on IU X-ray and 0.4340 on MIMIC-CXR, but 381 out of 575 (66.3%) IU outputs belonged to exact duplicate groups.Conclusions
No configuration performed consistently well across end points, and no single metric captured all forms of cross-domain degradation. The MedGemma stress test suggested that recurrence and label-traceability checks remain useful across architectures, although it was not a matched performance comparison. Cross-domain report generators should undergo bidirectional evaluation, final-text verification, and image-grounded human review before external validation or clinical deployment.
Related
- Calibration, Uncertainty, and Data Efficiency of Deep Learning Chest Radiograph Classifiers Under Cross-Dataset Distribution Shift: A Multi-Dataset Benchmark
- A Bidirectional Relationship Between Spontaneous Coronary Artery Dissection and Stress-Induced Cardiomyopathy in a Perimenopausal Woman: A Case Report.
- Bidirectional ventricular tachycardia unmasked by beta-blockade in pheochromocytoma presenting as hypertensive optic neuropathy: a case report and literature review.
- The ClimaSentAR dataset for evaluating cross-domain transferability of large language models in climate sentiment analysis
- Cross-Dataset Generalization in Phonocardiogram Murmur Classification: A Benchmark Study of Domain-Conditional Performance and the Role of Multi-Representation Fusion
BibTeX
@article{sun2026crossdomain,
title = {{Cross-Domain Chest Radiograph Report Generation Under Dataset Shift: Bidirectional Audit Study.}},
author = {Yishan Sun and Jun Jiang and Yu Chen},
journal = {JMIR medical informatics},
year = {2026},
month = oct,
doi = {10.2196/95888},
url = {https://europepmc.org/article/MED/42906076},
note = {CC BY},
}
RIS
TY - JOUR
TI - Cross-Domain Chest Radiograph Report Generation Under Dataset Shift: Bidirectional Audit Study.
AU - Yishan Sun
AU - Jun Jiang
AU - Yu Chen
PY - 2026
DA - 2026/10/21
JO - JMIR medical informatics
DO - 10.2196/95888
UR - https://europepmc.org/article/MED/42906076
AB - Background Cross-domain chest radiograph report generation may produce formally valid reports that are nevertheless repetitive, weakly case-specific, or missing clinically relevant findings. These failures can be missed when evaluation relies on a single language-overlap or clinical-label score.Objective This study aimed to evaluate cross-domain report generation with a bidirectional layered audit, characterize direction-specific and end point-specific failure profiles under a composite training-side control and 3 decoding policies, and explore whether selected audit dimensions remained informative for a contemporary medical vision language model.Methods A fixed R2Gen-style system was evaluated bidirectionally on 575 IU X-ray and 401 MIMIC-CXR (Medical Information Mart for Intensive Care Chest X-Ray) target records. The audit included surface validity, complete-report recurrence and concentration, a predefined prefix end point, lexical diversity, official Stanford CheXpert/NegBio labels, paired cluster-bootstrap inference, and blinded review by 2 medical trainees. The primary human audit included 210 reports from 70 randomly sampled cases, with blinded adjudication of unresolved primary end point disagreements. A 60-report targeted panel was used for failure-mode analysis. Separately, a fixed MedGemma 1.5 4B configuration generated reports for all 976 records.Results In MIMIC-CXR-to-IU X-ray transfer, out of 575 reports, ON (protocol-defined training-side control enabled)+B4 (beam search with a beam size of 4) reduced the predefined recurrent prefix from 575 (100%) to 8 (1.4%; Holm-adjusted P=.002). However, 569 (99.0%) ON+B4 outputs still belonged to exact duplicate groups. In IU X-ray-to-MIMIC-CXR transfer, the 14-category CheXpert/NegBio micro-F₁ was higher for ON+B4 than for OFF (protocol-defined training-side control disabled)+B4 (0.1687 vs 0.0744; Holm-adjusted P=.01). This apparent advantage was not broad-based: after cardiomegaly was excluded in a post hoc descriptive sensitivity analysis, the difference changed from +0.0944 to -0.0610. Text-level tracing showed that the original contrast was mainly driven by a labeler response to recurrent, semantically normal heart-size wording. Across the primary human-audit cells, major omissions occurred in 12 to 15 out of 35 reports, and major unsupported findings occurred in 10 to 16 of 35 reports, with no consistent configuration ordering observed. The exploratory MedGemma micro-F₁ was 0.4722 on IU X-ray and 0.4340 on MIMIC-CXR, but 381 out of 575 (66.3%) IU outputs belonged to exact duplicate groups.Conclusions No configuration performed consistently well across end points, and no single metric captured all forms of cross-domain degradation. The MedGemma stress test suggested that recurrence and label-traceability checks remain useful across architectures, although it was not a matched performance comparison. Cross-domain report generators should undergo bidirectional evaluation, final-text verification, and image-grounded human review before external validation or clinical deployment.
SN - 2291-9694
ER -