CLINES: Clinical LLM-based Information Extraction and Structuring Agent.
6 affiliations
- 1Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA.
- 2Department of Statistics and Data Science, Tsinghua University, Beijing, China.
- 3Division of Biostatistics, University of California Berkeley, Berkeley, California, USA.
- 4Harvard Medical School, Boston, Massachusetts, USA.
- 5Division of Interdisciplinary Medicine and Biotechnology, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA.
- 6Harvard T H Chan School of Public Health, Boston, Massachusetts, USA.
Abstract
Objective
Clinical narratives in electronic health records (EHRs) hold diagnostic, therapeutic and temporal information often missing from structured fields; manual chart review remains the de facto standard but is slow, costly and variable. We aimed to develop and evaluate Clinical LLM‑based Information Extraction and Structuring Agent (CLINES), a pipeline that extracts and normalises clinical concepts from free-text notes into ontology-grounded, auditable, schema-ready data without task-specific fine-tuning.Methods and analysis
We developed CLINES, a modular multistep pipeline: semantic chunking of long notes; extraction by reasoning-capable large language models (LLM); attribute assignment (assertion, value+unit); Unified Medical Language System normalisation; explicit and relative date resolution; and aggregation into an i2b2-style schema. Evaluation was conducted without task-specific supervised training on deidentified MIMIC-III, 4CE and CORAL (breast, pancreas) notes. Comparators include rule/lexicon systems, transformer encoders, single-prompt LLM baselines, alongside chain-of-thought LLMs and full-size clinical named entity recognition encoders (BERT-base, GatorTron-base). Outcomes were F1 for entity, assertion, value+unit and date processing, reported with 95% bootstrap CIs and paired-bootstrap, false discovery rate (FDR)-corrected tests. Inter-annotator agreement, hallucination taxonomy, cost and component ablations were also assessed.Results
Across all datasets and tasks, CLINES ranked first on every task-dataset combination. F1 ranges were 0.69-0.87 (entity), 0.84-0.93 (assertion), 0.77-0.90 (value+unit) and 0.73-0.79 (date, where gold annotations were available). Gains over the strongest single-prompt LLM were +0.21-0.38; transformer encoders trailed by +0.28-0.68 F1 on entity extraction. Performance remained stable as notes lengthened, while transformer baselines lost recall. Code field gains and most assertion/value comparisons were significant at FDR-adjusted p<0.05 on paired-bootstrap tests; mention-level inter-annotator F1 on a cross-annotated subset was 0.82 (assertion κ=0.85); a seven-class taxonomy on 700 stratified false positives found 54% of apparent false positives to be annotation gaps rather than true hallucinations.Conclusion
CLINES translates narrative EHR text into ontology-grounded, auditable, schema-ready data, offering a practical route to scale chart review-like extraction for cohort discovery and real-world evidence. CLINES substantially reduces but does not eliminate the need for manual review: the residual hallucination quantified here means human verification remains necessary for safety-critical use. CLINES is model agnostic-open and closed models can be substituted to meet cost, performance and privacy goals.
Related
- Structured information extraction from geoscientific literature with GeoExtractor
- Large Language Models for Structured Information Extraction from German Histopathology Reports - Hepatocellular Carcinoma
- DICD-IE: an expert-annotated benchmark for evaluating pretrained biomedical language models in dental implant literature and clinical knowledge extraction
- Development of a large language model clinical variable extraction pipeline for the study of lymphedema.
- Clinical Patterns, Indications, and Determinants of Third Molar Extraction among Dental Patients
BibTeX
@article{yang2026clines,
title = {{CLINES: Clinical LLM-based Information Extraction and Structuring Agent.}},
author = {Zongxin Yang and Hongyi Yuan and Raheel Sayeed and Amelia Tan and Enci Cai and Mohammed Moro and Xiudi Li and Huaiyuan Ying and Nicholas Brown and Griffin Weber and Sheng Yu and Isaac Kohane and Tianxi Cai},
journal = {BMJ digital health \& AI},
year = {2026},
month = oct,
doi = {10.1136/bmjdh-2026-000027},
url = {https://europepmc.org/article/MED/42948623},
note = {CC BY-NC},
}
RIS
TY - JOUR
TI - CLINES: Clinical LLM-based Information Extraction and Structuring Agent.
AU - Zongxin Yang
AU - Hongyi Yuan
AU - Raheel Sayeed
AU - Amelia Tan
AU - Enci Cai
AU - Mohammed Moro
AU - Xiudi Li
AU - Huaiyuan Ying
AU - Nicholas Brown
AU - Griffin Weber
AU - Sheng Yu
AU - Isaac Kohane
AU - Tianxi Cai
PY - 2026
DA - 2026/10/26
JO - BMJ digital health & AI
DO - 10.1136/bmjdh-2026-000027
UR - https://europepmc.org/article/MED/42948623
AB - Objective Clinical narratives in electronic health records (EHRs) hold diagnostic, therapeutic and temporal information often missing from structured fields; manual chart review remains the de facto standard but is slow, costly and variable. We aimed to develop and evaluate Clinical LLM‑based Information Extraction and Structuring Agent (CLINES), a pipeline that extracts and normalises clinical concepts from free-text notes into ontology-grounded, auditable, schema-ready data without task-specific fine-tuning.Methods and analysis We developed CLINES, a modular multistep pipeline: semantic chunking of long notes; extraction by reasoning-capable large language models (LLM); attribute assignment (assertion, value+unit); Unified Medical Language System normalisation; explicit and relative date resolution; and aggregation into an i2b2-style schema. Evaluation was conducted without task-specific supervised training on deidentified MIMIC-III, 4CE and CORAL (breast, pancreas) notes. Comparators include rule/lexicon systems, transformer encoders, single-prompt LLM baselines, alongside chain-of-thought LLMs and full-size clinical named entity recognition encoders (BERT-base, GatorTron-base). Outcomes were F1 for entity, assertion, value+unit and date processing, reported with 95% bootstrap CIs and paired-bootstrap, false discovery rate (FDR)-corrected tests. Inter-annotator agreement, hallucination taxonomy, cost and component ablations were also assessed.Results Across all datasets and tasks, CLINES ranked first on every task-dataset combination. F1 ranges were 0.69-0.87 (entity), 0.84-0.93 (assertion), 0.77-0.90 (value+unit) and 0.73-0.79 (date, where gold annotations were available). Gains over the strongest single-prompt LLM were +0.21-0.38; transformer encoders trailed by +0.28-0.68 F1 on entity extraction. Performance remained stable as notes lengthened, while transformer baselines lost recall. Code field gains and most assertion/value comparisons were significant at FDR-adjusted p<0.05 on paired-bootstrap tests; mention-level inter-annotator F1 on a cross-annotated subset was 0.82 (assertion κ=0.85); a seven-class taxonomy on 700 stratified false positives found 54% of apparent false positives to be annotation gaps rather than true hallucinations.Conclusion CLINES translates narrative EHR text into ontology-grounded, auditable, schema-ready data, offering a practical route to scale chart review-like extraction for cohort discovery and real-world evidence. CLINES substantially reduces but does not eliminate the need for manual review: the residual hallucination quantified here means human verification remains necessary for safety-critical use. CLINES is model agnostic-open and closed models can be substituted to meet cost, performance and privacy goals.
SN - 3049-575X
ER -