A Reproducible Decomposed Large Language Model Pipeline for Structured Abstraction of Multimodal Neuroimaging Reports: Development and Independent Validation in 1,669 Emergency Department Encounters for Dizziness and Vertigo
- 1Department of Neurology, Staten Island University Hospital, Northwell Health, Staten Island, NY, USA; Zucker School of Medicine at Hofstra/Northwell, Hempstead,
Abstract
Objective: To develop and independently validate a decomposed, modality-specific large language model (LLM) pipeline that abstracts structured variables from the free-text neuroimaging reports of a multimodal stroke workup, and to validate the resulting structured dataset for reuse in downstream research. Materials and Methods: A frozen pipeline (temperature 0) extracting 22 variables across CT head, CT angiography, CT perfusion, and MRI brain was applied to 1,669 ED encounters for dizziness or vertigo; 11 variables were pre-specified as primary (Tier 1). Reproducibility used three same-session runs and a nine-week repeat run; three frontier models (Claude Sonnet 4.6, GPT-5.4, Gemini 3.5 Flash) measured cross-model agreement (Cohen's/Fleiss' kappa). Accuracy was assessed in an independent 200-encounter cohort (150 random, 50 enriched for rare findings); 100 were double-read by two blinded reviewers with discordances adjudicated. Results: Test-retest reproducibility was high (Tier 1 Fleiss' kappa = 0.993; 99.8% exact) and stable nine weeks later (kappa = 0.992); inter-model agreement was kappa = 0.962. Against the adjudicated reference, pooled Tier 1 accuracy was 0.971 to 0.988 across models (Cohen's kappa 0.959 to 0.982), comparable to inter-reviewer agreement (kappa = 0.932). Accuracy was 0.980 to 0.983 in random and 0.955 to 0.977 in enriched encounters (agreement on model-flagged findings, not sensitivity). Discussion: A decomposed LLM pipeline produced reproducible, model-portable, stable, and accurate multimodal abstraction. Because schema design and validation are up-front costs, such pipelines scale to larger research cohorts. Conclusion: The quantified per-field error rates support reuse of this dataset with field-specific review, and such pipelines offer a practical path to large-scale, reproducible research datasets.
Related
- Cell-Type-Specific Transcriptomic Alterations in the Epileptic Prefrontal Cortex and Complementary Neuroimaging Findings in an Independent Cohort.
- Association of life course type 2 diabetes duration with adverse cardiac structure independent of current glycaemia in the British MRC National Survey of Health and Development Birth Cohort
- Cardiovascular Data Abstraction Centralization
- Camera-based HbA1c estimation using remote photoplethysmography: development, independent evaluation, and baseline comparator analysis
- Decomposed arterial stiffness and NfL–cognition association in older adults: The IGNITE study
BibTeX
@misc{shahani2026reproducible,
title = {{A Reproducible Decomposed Large Language Model Pipeline for Structured Abstraction of Multimodal Neuroimaging Reports: Development and Independent Validation in 1,669 Emergency Department Encounters for Dizziness and Vertigo}},
author = {J. Shahani and A. R. Ousta and S. Shaulov and Y. El-Sherif},
howpublished = {medRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.03.26367160},
url = {https://www.medrxiv.org/content/10.64898/2026.11.03.26367160v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - A Reproducible Decomposed Large Language Model Pipeline for Structured Abstraction of Multimodal Neuroimaging Reports: Development and Independent Validation in 1,669 Emergency Department Encounters for Dizziness and Vertigo
AU - J. Shahani
AU - A. R. Ousta
AU - S. Shaulov
AU - Y. El-Sherif
PY - 2026
DA - 2026/11/03
PB - medRxiv
DO - 10.64898/2026.11.03.26367160
UR - https://www.medrxiv.org/content/10.64898/2026.11.03.26367160v1
AB - Objective: To develop and independently validate a decomposed, modality-specific large language model (LLM) pipeline that abstracts structured variables from the free-text neuroimaging reports of a multimodal stroke workup, and to validate the resulting structured dataset for reuse in downstream research. Materials and Methods: A frozen pipeline (temperature 0) extracting 22 variables across CT head, CT angiography, CT perfusion, and MRI brain was applied to 1,669 ED encounters for dizziness or vertigo; 11 variables were pre-specified as primary (Tier 1). Reproducibility used three same-session runs and a nine-week repeat run; three frontier models (Claude Sonnet 4.6, GPT-5.4, Gemini 3.5 Flash) measured cross-model agreement (Cohen's/Fleiss' kappa). Accuracy was assessed in an independent 200-encounter cohort (150 random, 50 enriched for rare findings); 100 were double-read by two blinded reviewers with discordances adjudicated. Results: Test-retest reproducibility was high (Tier 1 Fleiss' kappa = 0.993; 99.8% exact) and stable nine weeks later (kappa = 0.992); inter-model agreement was kappa = 0.962. Against the adjudicated reference, pooled Tier 1 accuracy was 0.971 to 0.988 across models (Cohen's kappa 0.959 to 0.982), comparable to inter-reviewer agreement (kappa = 0.932). Accuracy was 0.980 to 0.983 in random and 0.955 to 0.977 in enriched encounters (agreement on model-flagged findings, not sensitivity). Discussion: A decomposed LLM pipeline produced reproducible, model-portable, stable, and accurate multimodal abstraction. Because schema design and validation are up-front costs, such pipelines scale to larger research cohorts. Conclusion: The quantified per-field error rates support reuse of this dataset with field-specific review, and such pipelines offer a practical path to large-scale, reproducible research datasets.
ER -