Large language model consensus for reliable research cohort construction from radiology reports: a retrospective cohort study
- 1New York University
Abstract
BackgroundLarge-scale retrospective studies often require researchers to adjudicate outcomes or cohort eligibility from unstructured clinical records. Manual review can provide reliable labels but is difficult to scale. We developed and evaluated a consensus workflow using multiple large language models (LLMs) to automatically assign high-confidence outcome labels while deferring ambiguous cases for manual review.
MethodsWe included 6,718 brain MRI reports from 5,856 patients. The expert-adjudicated reference set included 680 reports from 675 patients: 372 normal (age-appropriate without significant intracranial pathology), 90 white matter hyperintensity-only (WMH-only; isolated nonspecific WMHs potentially ambiguous for binary classification), and 218 abnormal (other intracranial pathology) reports. We evaluated proprietary-inclusive (Qwen3/Kimi-Linear/GPT-5) and open-source (Qwen3/Kimi-Linear/DeepSeek-R1) ensembles against individual LLMs. Ensembles assigned a normal or abnormal label only when all models agreed; all other reports were classified as uncertain and deferred for manual review. Individual LLMs assigned normal or abnormal predictions and classified WMH-only predictions as uncertain. In the full cohort, we compared the prevalence of active central nervous system (CNS)-related International Classification of Diseases, Tenth Revision (ICD-10) diagnoses across the normal, uncertain, and abnormal groups generated by each ensemble.
ResultsIn the expert-adjudicated reference set, both ensembles achieved 100% precision for automatically assigned normal and abnormal reports. GPT-5 was the strongest individual LLM but had lower precision than either ensemble, with normal and abnormal precision of 98.9% and 94.1%, respectively, and 17 expert-label mismatches versus none for the consensus ensembles. In the full cohort, 1.6% of reports classified as normal were associated with CNS-related ICD-10 diagnoses, compared with 29.7% of uncertain and 51.3% of abnormal reports for the proprietary-inclusive ensemble; corresponding rates were 1.6%, 31.1%, and 51.6% for the open-source ensemble.
ConclusionsRequiring unanimous agreement across multiple LLMs enabled high-precision automatic adjudication while directing ambiguous cases to manual review. This strategy may reduce the burden of manual adjudication and support scalable construction of high-purity research cohorts from unstructured clinical text.
Related
- Sycophancy in Large Language Models in Oral and Maxillofacial Radiology: The Impact of Misleading Context on Image- and Text-Based Diagnostic Scenarios
- Knowledge distillation-based lightweight generative large language models for standardized quality control of Chinese radiology reports
- Large Language Models in the Radiology Workflow: Current Applications, Challenges, and Future Perspectives.
- Enhancing medical education with AI: A comparative study of various large language models in radiology
- Natural Language Processing Based Solution for Labeling Brain Metastasis Identified in Radiology Reports
BibTeX
@misc{rui2026large,
title = {{Large language model consensus for reliable research cohort construction from radiology reports: a retrospective cohort study}},
author = {W. Rui and J. Ma and V. Stepanov and A. Bunnell and M. Lan and J. Chen and R. R. Patel and D. S. Novikov and E. Fieremans and Y. Shen},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.06.26364312},
url = {https://www.medrxiv.org/content/10.64898/2026.10.06.26364312v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Large language model consensus for reliable research cohort construction from radiology reports: a retrospective cohort study
AU - W. Rui
AU - J. Ma
AU - V. Stepanov
AU - A. Bunnell
AU - M. Lan
AU - J. Chen
AU - R. R. Patel
AU - D. S. Novikov
AU - E. Fieremans
AU - Y. Shen
PY - 2026
DA - 2026/10/06
PB - medRxiv
DO - 10.64898/2026.10.06.26364312
UR - https://www.medrxiv.org/content/10.64898/2026.10.06.26364312v1
AB - BackgroundLarge-scale retrospective studies often require researchers to adjudicate outcomes or cohort eligibility from unstructured clinical records. Manual review can provide reliable labels but is difficult to scale. We developed and evaluated a consensus workflow using multiple large language models (LLMs) to automatically assign high-confidence outcome labels while deferring ambiguous cases for manual review. MethodsWe included 6,718 brain MRI reports from 5,856 patients. The expert-adjudicated reference set included 680 reports from 675 patients: 372 normal (age-appropriate without significant intracranial pathology), 90 white matter hyperintensity-only (WMH-only; isolated nonspecific WMHs potentially ambiguous for binary classification), and 218 abnormal (other intracranial pathology) reports. We evaluated proprietary-inclusive (Qwen3/Kimi-Linear/GPT-5) and open-source (Qwen3/Kimi-Linear/DeepSeek-R1) ensembles against individual LLMs. Ensembles assigned a normal or abnormal label only when all models agreed; all other reports were classified as uncertain and deferred for manual review. Individual LLMs assigned normal or abnormal predictions and classified WMH-only predictions as uncertain. In the full cohort, we compared the prevalence of active central nervous system (CNS)-related International Classification of Diseases, Tenth Revision (ICD-10) diagnoses across the normal, uncertain, and abnormal groups generated by each ensemble. ResultsIn the expert-adjudicated reference set, both ensembles achieved 100% precision for automatically assigned normal and abnormal reports. GPT-5 was the strongest individual LLM but had lower precision than either ensemble, with normal and abnormal precision of 98.9% and 94.1%, respectively, and 17 expert-label mismatches versus none for the consensus ensembles. In the full cohort, 1.6% of reports classified as normal were associated with CNS-related ICD-10 diagnoses, compared with 29.7% of uncertain and 51.3% of abnormal reports for the proprietary-inclusive ensemble; corresponding rates were 1.6%, 31.1%, and 51.6% for the open-source ensemble. ConclusionsRequiring unanimous agreement across multiple LLMs enabled high-precision automatic adjudication while directing ambiguous cases to manual review. This strategy may reduce the burden of manual adjudication and support scalable construction of high-purity research cohorts from unstructured clinical text.
ER -