Half of alcohol, drug, and self-harm presentations cannot be identified in coded emergency department data: a diagnostic accuracy study of a large language model
- 1University of Edinburgh
Abstract
ObjectiveTo measure the accuracy of clinical coding, clinician review, and a locally deployed large language model (LLM) in identifying alcohol, drug, and self-harm involvement in emergency department (ED) attendances, and quantify prevalence.
DesignTwo-phase diagnostic accuracy study. In a validation week, the identification strategies were assessed against a conflict-adjudicated reference standard (n=2,256); the LLM was then applied to n=105,096 annual attendances at the same site.
SettingUK Type 1 Emergency Department treating patients [≥]16yrs.
Main outcome measuresPrevalence quantification compared with the reference standard; sensitivity, specificity, and balanced accuracy of each strategy; monthly identification rates and adjusted annual prevalence.
ResultsThe reference standard identified 12.1% of attendances as involving alcohol, drugs, or self-harm (coding 6.0%; clinician 10.0%, LLM 15.6%). LLM balanced accuracy matched or outperformed clinician review in all three domains (alcohol 0.942 v 0.930, p=0.635; drug 0.959 v 0.791, p<0.001; self-harm 0.982 v 0.908, p=0.004). Coding recorded 1.07 domains per identified patient against 1.32 in the reference standard. Adjusted annual prevalence corresponded to 12,890 domain involvements per year not identifiable in coded data. Subdomain classification found at least 81.6% of self-harm attendances required medical assessment for injury or overdose before psychiatric review.
ConclusionsClinical coding identified fewer than half of presentations involving alcohol, drugs, and self-harm and rarely captured co-occurring domains; under-recording was present across a full year. A locally deployed LLM generated more complete structured data from existing clinical text within NHS infrastructure, at a scale which is not feasible for manual review.
What is already known on this topicO_LICoded emergency department data underpin national surveillance, commissioning, research, and service planning, but the size and direction of their error for specific conditions have not been measured against a validated reference standard in an unselected population. C_LI
What this study addsO_LIIn a UK emergency department, clinical coding identified 6.0% of attendances as involving alcohol, drugs, or self-harm against an adjudicated reference-standard prevalence of 12.1%. C_LIO_LIIn blinded head-to-head comparison, a locally deployed, open-source large language model achieved similar or higher balanced accuracy than clinician review. Applied to 105,096 attendances over 12 months, it identified 14.3% of attendances against 4.1% by coding, in every month of the year. C_LI
How this study might affect research, practice, or policyO_LIPrevalence estimates and service planning based on coded emergency department data are likely to understate both the burden and the co-occurrence of these conditions. The majority (81.6%) of classified self-harm presentations required medical assessment for injury or overdose; only 18.4% did not, a figure directly relevant to mental health emergency-care streaming. Locally deployed open-source language models offer a scalable, governance-compatible method for identifying these presentations from existing free text. C_LI
Related
- An investigation of dental-related emergency department presentations and patient characteristics in Victoria, Australia.
- Impact of Increasing Ambient Temperatures on Emergency Department Trauma Presentations at a Level I Trauma Center.
- Ocular trauma presentations to a tertiary emergency department: Demographic characteristics, injury mechanisms, and clinical outcomes.
- Emergency Department Pediatric Early Warning System for Identifying Resuscitation-Defined Critical Illness in Children: A Prospective Diagnostic Classification Study in Vietnam
- CT-Referenced Diagnostic Performance and Complementary Error Patterns of Artificial Intelligence Versus a Single Emergency Physician in Chest Radiograph Interpretation: A Retrospective Paired Diagnostic Accuracy Study
BibTeX
@misc{humphries2026half,
title = {{Half of alcohol, drug, and self-harm presentations cannot be identified in coded emergency department data: a diagnostic accuracy study of a large language model}},
author = {C. Humphries and J. Brett and F. Gruber and E. James and T. I. McKendrick and K. C. McNairn and A. Miell and R. O'Brien and F. Rahman and L. Schölin and M. Stewart and A. Casey},
howpublished = {medRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.26.26361443},
url = {https://www.medrxiv.org/content/10.64898/2026.08.26.26361443v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Half of alcohol, drug, and self-harm presentations cannot be identified in coded emergency department data: a diagnostic accuracy study of a large language model
AU - C. Humphries
AU - J. Brett
AU - F. Gruber
AU - E. James
AU - T. I. McKendrick
AU - K. C. McNairn
AU - A. Miell
AU - R. O'Brien
AU - F. Rahman
AU - L. Schölin
AU - M. Stewart
AU - A. Casey
PY - 2026
DA - 2026/08/26
PB - medRxiv
DO - 10.64898/2026.08.26.26361443
UR - https://www.medrxiv.org/content/10.64898/2026.08.26.26361443v1
AB - ObjectiveTo measure the accuracy of clinical coding, clinician review, and a locally deployed large language model (LLM) in identifying alcohol, drug, and self-harm involvement in emergency department (ED) attendances, and quantify prevalence. DesignTwo-phase diagnostic accuracy study. In a validation week, the identification strategies were assessed against a conflict-adjudicated reference standard (n=2,256); the LLM was then applied to n=105,096 annual attendances at the same site. SettingUK Type 1 Emergency Department treating patients [≥]16yrs. Main outcome measuresPrevalence quantification compared with the reference standard; sensitivity, specificity, and balanced accuracy of each strategy; monthly identification rates and adjusted annual prevalence. ResultsThe reference standard identified 12.1% of attendances as involving alcohol, drugs, or self-harm (coding 6.0%; clinician 10.0%, LLM 15.6%). LLM balanced accuracy matched or outperformed clinician review in all three domains (alcohol 0.942 v 0.930, p=0.635; drug 0.959 v 0.791, p<0.001; self-harm 0.982 v 0.908, p=0.004). Coding recorded 1.07 domains per identified patient against 1.32 in the reference standard. Adjusted annual prevalence corresponded to 12,890 domain involvements per year not identifiable in coded data. Subdomain classification found at least 81.6% of self-harm attendances required medical assessment for injury or overdose before psychiatric review. ConclusionsClinical coding identified fewer than half of presentations involving alcohol, drugs, and self-harm and rarely captured co-occurring domains; under-recording was present across a full year. A locally deployed LLM generated more complete structured data from existing clinical text within NHS infrastructure, at a scale which is not feasible for manual review. What is already known on this topicO_LICoded emergency department data underpin national surveillance, commissioning, research, and service planning, but the size and direction of their error for specific conditions have not been measured against a validated reference standard in an unselected population. C_LI What this study addsO_LIIn a UK emergency department, clinical coding identified 6.0% of attendances as involving alcohol, drugs, or self-harm against an adjudicated reference-standard prevalence of 12.1%. C_LIO_LIIn blinded head-to-head comparison, a locally deployed, open-source large language model achieved similar or higher balanced accuracy than clinician review. Applied to 105,096 attendances over 12 months, it identified 14.3% of attendances against 4.1% by coding, in every month of the year. C_LI How this study might affect research, practice, or policyO_LIPrevalence estimates and service planning based on coded emergency department data are likely to understate both the burden and the co-occurrence of these conditions. The majority (81.6%) of classified self-harm presentations required medical assessment for injury or overdose; only 18.4% did not, a figure directly relevant to mental health emergency-care streaming. Locally deployed open-source language models offer a scalable, governance-compatible method for identifying these presentations from existing free text. C_LI
ER -