Beyond ICD Codes: Fine-Tuning LLMs for In-Hospital Cardiac Arrest Identification from EHR Notes
- 1Cedars-Sinai Medical Center
Abstract
BackgroundIdentification of in-hospital cardiac arrest (IHCA) through manual chart abstraction is the gold standard, but its time-consuming and resource-intensive, which naturally limits its practicality. Diagnostic codes provide an accessible automated alternative, but prior work has shown this method of extraction has poor sensitivity and poor positive predictive value. More accurate and scalable automated methods are needed to support IHCA surveillance and research.
MethodsWe developed large language model (LLM)-based systems that incorporated physician-defined clinical knowledge through guideline-enriched prompting, supervised finetuning, and a self-correction procedure. The systems identified IHCA events and their locations from electronic health record notes. Performance was evaluated at the encounter level against physician-adjudicated chart review and compared with ICD-code-based identification.
ResultsCompared with an ICD-code-based algorithm (precision 0.68, recall 0.91, F1 score 0.78), the supervised fine-tuned LLM offered substantially better performance for IHCA identification on a curated evaluation corpus, reaching a precision of 0.88, a recall of 1.00, and an F1 score of 0.94. When tested on a larger, unselected validation cohort of 45,525 encounters, a separate LLM configuration--relying on guideline-enriched prompting and a self-correction procedure rather than fine-tuning--achieved a precision of 0.79 for encounter-level IHCA identification, meaning most encounters flagged by the model were genuine IHCA cases. The fine-tuned LLM underperformed on this cohort due to its inability to self-correct its decisions, an issue that can be mitigated by having a separate model perform the correction step.
ConclusionExplicit integration of clinical knowledge through prompting and supervised fine-tuning enabled accurate identification of IHCA events and locations from routine clinical documentation. In a two-pass workflow, these systems could prioritize encounters for clinician adjudication, reduce manual screening requirements, and support more scalable IHCA surveillance and registry development.
Related
- In-Hospital Mortality Trends Among Hospitalized Adults With Cardiac Arrest, 2016-2022
- Favourable discharge after cardiac arrest across APACHE IVa-predicted hospital mortality: a multicentre observational study
- Prognostic Impact of Transient Return of Spontaneous Circulation According to Low-Flow Time in Patients with Out-of-Hospital Cardiac Arrest Undergoing Extracorporeal Cardiopulmonary Resuscitation: A Nationwide Cohort Study
- Post In-Hospital Cardiac Arrest Simulation to Improve Medical Decision Making and Communication in Internal Medicine Residents
- The impact of the SYNTAX score on the incidence of in-hospital cardiac arrest in patients with acute coronary syndrome undergoing percutaneous coronary intervention: a retrospective cohort study
BibTeX
@misc{weissenbacher2026beyond,
title = {{Beyond ICD Codes: Fine-Tuning LLMs for In-Hospital Cardiac Arrest Identification from EHR Notes}},
author = {D. Weissenbacher and J. Vo and A. Uy-Evanado and K. Reinier and H. Chugh and V. Kadiyala and K. OCOnnor and S. S. Chugh and G. Gonzalez-Hernandez},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.24.26363973},
url = {https://www.medrxiv.org/content/10.64898/2026.09.24.26363973v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Beyond ICD Codes: Fine-Tuning LLMs for In-Hospital Cardiac Arrest Identification from EHR Notes
AU - D. Weissenbacher
AU - J. Vo
AU - A. Uy-Evanado
AU - K. Reinier
AU - H. Chugh
AU - V. Kadiyala
AU - K. OCOnnor
AU - S. S. Chugh
AU - G. Gonzalez-Hernandez
PY - 2026
DA - 2026/09/24
PB - medRxiv
DO - 10.64898/2026.09.24.26363973
UR - https://www.medrxiv.org/content/10.64898/2026.09.24.26363973v1
AB - BackgroundIdentification of in-hospital cardiac arrest (IHCA) through manual chart abstraction is the gold standard, but its time-consuming and resource-intensive, which naturally limits its practicality. Diagnostic codes provide an accessible automated alternative, but prior work has shown this method of extraction has poor sensitivity and poor positive predictive value. More accurate and scalable automated methods are needed to support IHCA surveillance and research. MethodsWe developed large language model (LLM)-based systems that incorporated physician-defined clinical knowledge through guideline-enriched prompting, supervised finetuning, and a self-correction procedure. The systems identified IHCA events and their locations from electronic health record notes. Performance was evaluated at the encounter level against physician-adjudicated chart review and compared with ICD-code-based identification. ResultsCompared with an ICD-code-based algorithm (precision 0.68, recall 0.91, F1 score 0.78), the supervised fine-tuned LLM offered substantially better performance for IHCA identification on a curated evaluation corpus, reaching a precision of 0.88, a recall of 1.00, and an F1 score of 0.94. When tested on a larger, unselected validation cohort of 45,525 encounters, a separate LLM configuration--relying on guideline-enriched prompting and a self-correction procedure rather than fine-tuning--achieved a precision of 0.79 for encounter-level IHCA identification, meaning most encounters flagged by the model were genuine IHCA cases. The fine-tuned LLM underperformed on this cohort due to its inability to self-correct its decisions, an issue that can be mitigated by having a separate model perform the correction step. ConclusionExplicit integration of clinical knowledge through prompting and supervised fine-tuning enabled accurate identification of IHCA events and locations from routine clinical documentation. In a two-pass workflow, these systems could prioritize encounters for clinician adjudication, reduce manual screening requirements, and support more scalable IHCA surveillance and registry development.
ER -