Evaluating Large Language Model Diagnostic Classification in Neuropsychology
Abstract
Objective: Large language models (LLMs) have demonstrated promise in healthcare for clinical documentation and decision support. Much of the information in electronic medical records, however, is structured or clinician-synthesized documentation. There is less certainty about the capability of LLMs in devising clinically appropriate diagnostic hypotheses from unstructured data such as a patient interview. Method: Deidentified clinical interview transcripts from 32 older adults referred for neuropsychological evaluation of memory complaints were submitted to 10 commercially available LLMs using an institutionally approved platform with a standardized prompt. Models classified each case using four categories: normal, mild cognitive impairment (MCI), dementia, or uncertain. Model classifications were compared with clinician reference diagnoses based on neuropsychological evaluation. Results: Model-inferred and reference diagnoses showed exact agreement rates from 18.8% to 56.2% with poor to slight chance-corrected agreement (Cohen’s κ = −.178 to .196). Classification errors across models were more often overclassification rather than underclassification. Using dichotomous cognitive status (i.e., normal versus impaired), seven of ten models showed sensitivity over 90% for identifying impairment, but specificity ranged from 0 to 60%. Workflow response time varied by model and was not associated with accuracy. Conclusions: Using unstructured and unannotated clinical interview data, general-purpose LLMs showed limited, model-dependent agreement with clinician diagnoses and a tendency to overclassify impairment. LLMs warrant further evaluation as tools that support clinical workflows, with revised prompting strategies and systematic efforts to improve classification.
Related
- Evaluating Large Language Models as Tools to Navigate Researchers in Rapidly Evolving Research Landscapes: A Case Study in Cancer Drug Response Prediction
- Evaluating a Large Language Model for Emergency Triage Using Synthetic Clinical Dialogues: Toward a Multimodal Clinical Safety Framework
- Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
- Sycophancy in Large Language Models in Oral and Maxillofacial Radiology: The Impact of Misleading Context on Image- and Text-Based Diagnostic Scenarios
- Expert-Guided Visual Correction for Characterizing Diagnostic Performance and Error Patterns of Multimodal Large Language Models Using Periodontal In-Service Examination Images
BibTeX
@misc{hachem2026evaluating,
title = {{Evaluating Large Language Model Diagnostic Classification in Neuropsychology}},
author = {Anthony Hachem and Chloe Chui and Elliot T. Morgan and Ashika S. Bakre and Summer Rolin and Anthony Rios and Jeremy Davis},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/6pmca_v1},
url = {https://osf.io/preprints/psyarxiv/6pmca_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evaluating Large Language Model Diagnostic Classification in Neuropsychology
AU - Anthony Hachem
AU - Chloe Chui
AU - Elliot T. Morgan
AU - Ashika S. Bakre
AU - Summer Rolin
AU - Anthony Rios
AU - Jeremy Davis
PY - 2026
DA - 2026/10/06
PB - PsyArXiv
DO - 10.31234/osf.io/6pmca_v1
UR - https://osf.io/preprints/psyarxiv/6pmca_v1/
AB - Objective: Large language models (LLMs) have demonstrated promise in healthcare for clinical documentation and decision support. Much of the information in electronic medical records, however, is structured or clinician-synthesized documentation. There is less certainty about the capability of LLMs in devising clinically appropriate diagnostic hypotheses from unstructured data such as a patient interview. Method: Deidentified clinical interview transcripts from 32 older adults referred for neuropsychological evaluation of memory complaints were submitted to 10 commercially available LLMs using an institutionally approved platform with a standardized prompt. Models classified each case using four categories: normal, mild cognitive impairment (MCI), dementia, or uncertain. Model classifications were compared with clinician reference diagnoses based on neuropsychological evaluation. Results: Model-inferred and reference diagnoses showed exact agreement rates from 18.8% to 56.2% with poor to slight chance-corrected agreement (Cohen’s κ = −.178 to .196). Classification errors across models were more often overclassification rather than underclassification. Using dichotomous cognitive status (i.e., normal versus impaired), seven of ten models showed sensitivity over 90% for identifying impairment, but specificity ranged from 0 to 60%. Workflow response time varied by model and was not associated with accuracy. Conclusions: Using unstructured and unannotated clinical interview data, general-purpose LLMs showed limited, model-dependent agreement with clinician diagnoses and a tendency to overclassify impairment. LLMs warrant further evaluation as tools that support clinical workflows, with revised prompting strategies and systematic efforts to improve classification.
ER -