Richer database context improves LLM-generated EHR phenotyping algorithms
Abstract
Objective: To evaluate large language model (LLM) generation of type 2 diabetes (T2DM) phenotyping SQL and the effect of database implementation context. Materials and Methods: Eight publicly accessible LLM-based systems generated SQL under progressively richer prompts using synthetic and University of North Carolina (UNC) Health electronic health record (EHR) data, both based on the Observational Medical Outcomes Partnership (OMOP) model. Outputs were compared with the Electronic Medical Records and Genomics (eMERGE) T2DM reference phenotype using SQL executability, F1, positive predictive value (PPV), and recall. Results: Fewer than 55% of limited-context synthetic outputs and no limited-context UNC EHR outputs executed without modification. Detailed codebooks (L3) increased executability to 81.3% in synthetic OMOP and 100% in UNC EHR; adding sample rows (L4) increased synthetic executability to 100%. Limited-context recall was as low as 0.01, whereas high-context outputs achieved recall [≥]0.99 at synthetic L4 and >0.90 at UNC L3, although UNC PPV remained substantially lower than recall. At the highest available context levels, Beta F1 ranged from 0.72-0.99 in synthetic OMOP and 0.57-0.68 in UNC EHR; all UNC models exceeded the diagnosis-code baseline (F1=0.54). The top synthetic performer fell from F1=0.99 to 0.64 on UNC EHR. Discussion: Richer database context was associated with more consistent SQL executability and recall, while model rankings varied across datasets. Conclusion: LLMs can generate clinically meaningful T2DM phenotyping SQL when provided with sufficient database context, but limited PPV in real-world data underscores the need for local validation.
Related
- Validation of Algorithms to Identify Small or Large for Gestational Age in the Korean Nationwide Healthcare Database
- Developing an OMOP-Standardized Prostate Cancer Database and Improving Data Quality Using NLP and PSA-Based Algorithms
- Continuous Glucose Monitoring for Metabolic Phenotyping and Precision Nutrition: Current Evidence, Clinical Applications and Future Perspectives
- Four-Year Longitudinal Phenotyping of Post-Acute Sequelae of SARS-CoV-2 Infection (PASC): A Single-Centre Cohort Study
- The liver–muscle axis in metabolic dysfunction-associated steatotic liver disease: a narrative review and conceptual framework for body-composition phenotyping
BibTeX
@misc{ye2026richer,
title = {{Richer database context improves LLM-generated EHR phenotyping algorithms}},
author = {Mengshuo Ye and Harlin Lee and Selen Gizlice and Madeline Tobias and Shawn O’Neil and Emily Pfaff and Stanley C Ahalt},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.27.26366654},
url = {https://www.medrxiv.org/content/10.64898/2026.10.27.26366654v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Richer database context improves LLM-generated EHR phenotyping algorithms
AU - Mengshuo Ye
AU - Harlin Lee
AU - Selen Gizlice
AU - Madeline Tobias
AU - Shawn O’Neil
AU - Emily Pfaff
AU - Stanley C Ahalt
PY - 2026
DA - 2026/10/27
PB - medRxiv
DO - 10.64898/2026.10.27.26366654
UR - https://www.medrxiv.org/content/10.64898/2026.10.27.26366654v1
AB - Objective: To evaluate large language model (LLM) generation of type 2 diabetes (T2DM) phenotyping SQL and the effect of database implementation context. Materials and Methods: Eight publicly accessible LLM-based systems generated SQL under progressively richer prompts using synthetic and University of North Carolina (UNC) Health electronic health record (EHR) data, both based on the Observational Medical Outcomes Partnership (OMOP) model. Outputs were compared with the Electronic Medical Records and Genomics (eMERGE) T2DM reference phenotype using SQL executability, F1, positive predictive value (PPV), and recall. Results: Fewer than 55% of limited-context synthetic outputs and no limited-context UNC EHR outputs executed without modification. Detailed codebooks (L3) increased executability to 81.3% in synthetic OMOP and 100% in UNC EHR; adding sample rows (L4) increased synthetic executability to 100%. Limited-context recall was as low as 0.01, whereas high-context outputs achieved recall [≥]0.99 at synthetic L4 and >0.90 at UNC L3, although UNC PPV remained substantially lower than recall. At the highest available context levels, Beta F1 ranged from 0.72-0.99 in synthetic OMOP and 0.57-0.68 in UNC EHR; all UNC models exceeded the diagnosis-code baseline (F1=0.54). The top synthetic performer fell from F1=0.99 to 0.64 on UNC EHR. Discussion: Richer database context was associated with more consistent SQL executability and recall, while model rankings varied across datasets. Conclusion: LLMs can generate clinically meaningful T2DM phenotyping SQL when provided with sufficient database context, but limited PPV in real-world data underscores the need for local validation.
ER -