Assessing AI and Neurologist Diagnostic Reasoning Against Neuropathological Ground Truth
- 1Massachusetts General Hospital
Abstract
BACKGROUNDAccurate differential diagnosis of complex neurological disorders remains challenging due to overlapping clinical features and heterogeneous disease presentations. Although large language models (LLMs) show promise in clinical reasoning, prior studies have benchmarked performance against clinician consensus rather than biological ground truth. Evaluation of diagnostic AI in neurology requires benchmarks grounded in neuropathological findings.
METHODSWe introduce NeuroBench, a curated benchmark of complex neurological cases with neuropathologically confirmed reference-standard diagnoses, and DIAGNO, a confidence-aware LLM-based system for neurological diagnosis. NeuroBench comprises 791 retrospective case summaries and corresponding autopsy-confirmed diagnoses across three independent health systems: 500 cases from the Massachusetts General Hospital (MGH) Brain Cutting Conference; 200 cases from the Mount Sinai Health System; and 91 cases from the University of Texas Health Science Center at San Antonio (UTHSA). Across all cases, DIAGNO generated top-3 differential diagnoses, employing retrieval-augmented generation for lower-confidence cases. In a subset of 203 complex MGH cases, performance was assessed by three independent blinded adjudicators who evaluated both DIAGNO and neurologists against neuropathological ground truth.
RESULTSNeuroBench encompassed 59 unique neuropathological diagnoses, spanning conditions including cerebrovascular disease, brain tumors, neurological infections, and various neurodegenerative and inflammatory disorders. DIAGNO achieved 83.8% top-3 accuracy across the 500 MGH cases, 76.5% accuracy across the 200 Mount Sinai cases, and 92.3% accuracy across the 91 UTHSA cases. In the subset of 203 MGH cases, DIAGNO achieved higher top-3 accuracy (0.67 versus 0.63) and taxonomy-level accuracy (0.74 versus 0.67) than neurologists. In cases of disagreement, DIAGNO was more often correct than neurologists (29 versus 19 cases). Diagnostic concordance between DIAGNO and neurologists was high (90% agreement in top-3 predictions). In a real-world evaluation on eight cases from Mass General Brigham, neurologists rated DIAGNOs reasoning favorably (mean 4.03/5) across multiple dimensions.
CONCLUSIONSNeuroBench establishes neuropathological confirmation as a reference standard for evaluating diagnostic AI in neurology, moving beyond clinician-based benchmarking to redefine the ceiling of diagnostic accuracy. Evaluated against this standard, DIAGNO achieved diagnostic performance comparable to neurologists and received favorable clinician ratings in real-world applications, supporting its potential as a clinical decision support tool in neurology.
Related
- Diagnostic accuracy of neuropathological assessment across different diagnostic settings for high-grade gliomas: the added value of DNA methylation profiling
- Information theory and diagnostic reasoning in primary care: a framework for navigating uncertainty.
- Stroke-Heart Syndrome: Mechanisms, Temporal Phenotypes, and Diagnostic Reasoning.
- Comparison of CT-based migration analysis against radiostereometric analysis for assessing two-year migration in an uncemented medial pivot tibial knee implant : clinical implications from a prospective safety study
- Decompression Sickness Myelopathy: Case Report and Acute Management Review for the Neurologist
BibTeX
@misc{leng2026assessing,
title = {{Assessing AI and Neurologist Diagnostic Reasoning Against Neuropathological Ground Truth}},
author = {Y. Leng and A. Noori and Y. He and A. Favaro and J. R. Dickson and A. Serrano-Pozo and M. Avetisyan and D. Rodriguez and E. S. Rosenberg and D. H. Oakley and V. S. Khurana and J. Maslinski and B. S. Glicksberg and M. Syed and P. E. Kubon and J. Walker and C. K. Suemoto and S. Seshadri and D. A. Clifton and M. Zitnik and B. T. Hyman and M. P. Frosch and S. Das},
howpublished = {medRxiv},
year = {2026},
month = jul,
doi = {10.64898/2026.07.07.26356930},
url = {https://www.medrxiv.org/content/10.64898/2026.07.07.26356930v2},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Assessing AI and Neurologist Diagnostic Reasoning Against Neuropathological Ground Truth
AU - Y. Leng
AU - A. Noori
AU - Y. He
AU - A. Favaro
AU - J. R. Dickson
AU - A. Serrano-Pozo
AU - M. Avetisyan
AU - D. Rodriguez
AU - E. S. Rosenberg
AU - D. H. Oakley
AU - V. S. Khurana
AU - J. Maslinski
AU - B. S. Glicksberg
AU - M. Syed
AU - P. E. Kubon
AU - J. Walker
AU - C. K. Suemoto
AU - S. Seshadri
AU - D. A. Clifton
AU - M. Zitnik
AU - B. T. Hyman
AU - M. P. Frosch
AU - S. Das
PY - 2026
DA - 2026/07/07
PB - medRxiv
DO - 10.64898/2026.07.07.26356930
UR - https://www.medrxiv.org/content/10.64898/2026.07.07.26356930v2
AB - BACKGROUNDAccurate differential diagnosis of complex neurological disorders remains challenging due to overlapping clinical features and heterogeneous disease presentations. Although large language models (LLMs) show promise in clinical reasoning, prior studies have benchmarked performance against clinician consensus rather than biological ground truth. Evaluation of diagnostic AI in neurology requires benchmarks grounded in neuropathological findings. METHODSWe introduce NeuroBench, a curated benchmark of complex neurological cases with neuropathologically confirmed reference-standard diagnoses, and DIAGNO, a confidence-aware LLM-based system for neurological diagnosis. NeuroBench comprises 791 retrospective case summaries and corresponding autopsy-confirmed diagnoses across three independent health systems: 500 cases from the Massachusetts General Hospital (MGH) Brain Cutting Conference; 200 cases from the Mount Sinai Health System; and 91 cases from the University of Texas Health Science Center at San Antonio (UTHSA). Across all cases, DIAGNO generated top-3 differential diagnoses, employing retrieval-augmented generation for lower-confidence cases. In a subset of 203 complex MGH cases, performance was assessed by three independent blinded adjudicators who evaluated both DIAGNO and neurologists against neuropathological ground truth. RESULTSNeuroBench encompassed 59 unique neuropathological diagnoses, spanning conditions including cerebrovascular disease, brain tumors, neurological infections, and various neurodegenerative and inflammatory disorders. DIAGNO achieved 83.8% top-3 accuracy across the 500 MGH cases, 76.5% accuracy across the 200 Mount Sinai cases, and 92.3% accuracy across the 91 UTHSA cases. In the subset of 203 MGH cases, DIAGNO achieved higher top-3 accuracy (0.67 versus 0.63) and taxonomy-level accuracy (0.74 versus 0.67) than neurologists. In cases of disagreement, DIAGNO was more often correct than neurologists (29 versus 19 cases). Diagnostic concordance between DIAGNO and neurologists was high (90% agreement in top-3 predictions). In a real-world evaluation on eight cases from Mass General Brigham, neurologists rated DIAGNOs reasoning favorably (mean 4.03/5) across multiple dimensions. CONCLUSIONSNeuroBench establishes neuropathological confirmation as a reference standard for evaluating diagnostic AI in neurology, moving beyond clinician-based benchmarking to redefine the ceiling of diagnostic accuracy. Evaluated against this standard, DIAGNO achieved diagnostic performance comparable to neurologists and received favorable clinician ratings in real-world applications, supporting its potential as a clinical decision support tool in neurology.
ER -