Extracting Symptoms of Psychotic Disorders from Clinical Notes using Natural Language Processing.
Abstract
Large Language Models (LLMs) are proposed as tools for high-throughput, deep phenotyping of psychiatric disorders. Applied to electronic health records, LLMs could in principle extract patient symptoms, outcome trajectories, risk factors, and treatment history at scale and these, when combined with increasingly available biological data, such as genomics or neuroimaging, could provide powerful resources for health research. Although proof-of-principle LLM-based symptom extractions have been carried out for some medical conditions, the heterogeneous nature of psychiatric disorders, including schizophrenia, requires extensive domain-specific evaluations. Here, we evaluate 14 general-purpose LLMs for extracting eight symptoms of schizophrenia from clinical summaries in a severe mental illness cohort (N = 704). Performance across symptoms was poor to moderate (macro F1 = 0.500 - 0.647), with positive symptoms more accurately extracted than negative symptoms. Few-shot prompting, a common strategy for improving LLM task performance, did not significantly improve these results. Nevertheless, LLM-predicted and gold-standard positive symptoms demonstrated comparable associations with clinical variables in regression analyses. Individual-level extraction errors attenuated group-level associations but did so unevenly across symptom domains. This indicates that general-purpose LLMs may not be able to extract psychosis-related phenotypes from clinical summaries with the quality required for clinical use but might nevertheless be useful for exploratory research on large cohorts. However, closing the performance gap between positive and negative symptom extractions seems essential groundwork to prevent a systematic bias in any LLM-derived characterisations of psychotic symptoms.
Related
- Extracting Symptoms Relevant to Geriatric Syndromes from Japanese Pharmaceutical Care Records: A Natural Language Processing Analysis
- Prospective Bidirectional Associations Between Core Schemas, Attenuated Psychotic Symptoms, and Functioning in Youth at Clinical High Risk for Psychosis
- A twin study of genetic and environmental aetiologies of associations between sensory processing sensitivity and psychotic-like experiences
- Long-Term Course of Loneliness in Psychotic Disorders
- Effect of ambient temperature and thermal comfort indices on emergency visits for psychotic and mood disorders: a case-crossover study in a French suburban area
BibTeX
@misc{lock2026extracting,
title = {{Extracting Symptoms of Psychotic Disorders from Clinical Notes using Natural Language Processing.}},
author = {S. K. Lock and J. Boisson and L. M. Evans and Y. Shi and D. B. Kappel and J. T. R. Walters and M. C. O'Donovan and S. E. Legge and J. Camacho-Collados and A. F. Pardinas},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.16.26363090},
url = {https://www.medrxiv.org/content/10.64898/2026.09.16.26363090v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Extracting Symptoms of Psychotic Disorders from Clinical Notes using Natural Language Processing.
AU - S. K. Lock
AU - J. Boisson
AU - L. M. Evans
AU - Y. Shi
AU - D. B. Kappel
AU - J. T. R. Walters
AU - M. C. O'Donovan
AU - S. E. Legge
AU - J. Camacho-Collados
AU - A. F. Pardinas
PY - 2026
DA - 2026/09/16
PB - medRxiv
DO - 10.64898/2026.09.16.26363090
UR - https://www.medrxiv.org/content/10.64898/2026.09.16.26363090v1
AB - Large Language Models (LLMs) are proposed as tools for high-throughput, deep phenotyping of psychiatric disorders. Applied to electronic health records, LLMs could in principle extract patient symptoms, outcome trajectories, risk factors, and treatment history at scale and these, when combined with increasingly available biological data, such as genomics or neuroimaging, could provide powerful resources for health research. Although proof-of-principle LLM-based symptom extractions have been carried out for some medical conditions, the heterogeneous nature of psychiatric disorders, including schizophrenia, requires extensive domain-specific evaluations. Here, we evaluate 14 general-purpose LLMs for extracting eight symptoms of schizophrenia from clinical summaries in a severe mental illness cohort (N = 704). Performance across symptoms was poor to moderate (macro F1 = 0.500 - 0.647), with positive symptoms more accurately extracted than negative symptoms. Few-shot prompting, a common strategy for improving LLM task performance, did not significantly improve these results. Nevertheless, LLM-predicted and gold-standard positive symptoms demonstrated comparable associations with clinical variables in regression analyses. Individual-level extraction errors attenuated group-level associations but did so unevenly across symptom domains. This indicates that general-purpose LLMs may not be able to extract psychosis-related phenotypes from clinical summaries with the quality required for clinical use but might nevertheless be useful for exploratory research on large cohorts. However, closing the performance gap between positive and negative symptom extractions seems essential groundwork to prevent a systematic bias in any LLM-derived characterisations of psychotic symptoms.
ER -