"I Know Where to Look," But Does the LLM? Charting the Gaps Between Clinical Expert Needs and Unstructured Data Abstraction Tools
Abstract
Clinical data abstraction, the process of distilling structured information from patient records, plays a key role in advancing knowledge about diseases such as cancer. Information extraction (IE) with large language models (LLMs) could accelerate this process, but it is unclear whether current frameworks effectively support clinical researchers without AI expertise. To address this, we co-designed an interactive LLM-based abstraction system called Libretto with seven cancer research teams, then evaluated the system's ability to help them answer real-world research questions. We found that while clinicians knew where and how to annotate complex concepts in patient notes, in twelve of fourteen tasks they faced barriers to replicating those intuitions with LLMs. Contextual note reliability judgments, difficulties in steering vibe-coded prompts, and inflexible evaluation strategies necessitated fundamental changes to the IE workflow. Our results highlight open problems for HCI research to bridge the gaps between AI data work tools and clinical users' needs.
Related
- Bern Sleep-Wake Registry: temporal extraction of comorbidities from unstructured clinical text using a large language model
- Privacy-Preserving Large Language Model Deployment for Oncology Registry Abstraction: Structure-Aware Evaluation in a Real-World Clinical Setting
- Where did it go? How Information Management Practices Impact Rigor and Reproducibility in Collaborative Clinical and Translational Research
- Global perspectives on dietary management of small intestinal bacterial overgrowth (SIBO): a systematic review of clinical guidelines, consensus statements and expert recommendations
- Nutrient Calculation Accuracy, Expert-Rated Clinical Appropriateness and Safety Indicators, and Reproducibility of Large Language Model-Generated Diet Plans for Polyendocrine Metabolic Ovarian Syndrome: A Real-Case-Based Evaluation
BibTeX
@misc{sivaraman2026know,
title = {{"I Know Where to Look," But Does the LLM? Charting the Gaps Between Clinical Expert Needs and Unstructured Data Abstraction Tools}},
author = {Venkatesh Sivaraman and Rigney Turnham and George Bonano and Nevin Aresh and Renumathy Dhanasekaran and Margaret Guo and Sindhu Kubendran and Olivia Lin and Jonathan D Louie and Kristan Olazo and Jeanne Shen and Harish Vasudevan and Jeanette Wong and Emily Alsentzer and Jason A Fries and Anobel Odisho and John Gordan and Jean Feng and Julian C Hong},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.19318},
eprint = {2609.19318},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.19318},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - "I Know Where to Look," But Does the LLM? Charting the Gaps Between Clinical Expert Needs and Unstructured Data Abstraction Tools
AU - Venkatesh Sivaraman
AU - Rigney Turnham
AU - George Bonano
AU - Nevin Aresh
AU - Renumathy Dhanasekaran
AU - Margaret Guo
AU - Sindhu Kubendran
AU - Olivia Lin
AU - Jonathan D Louie
AU - Kristan Olazo
AU - Jeanne Shen
AU - Harish Vasudevan
AU - Jeanette Wong
AU - Emily Alsentzer
AU - Jason A Fries
AU - Anobel Odisho
AU - John Gordan
AU - Jean Feng
AU - Julian C Hong
PY - 2026
DA - 2026/09/16
PB - arXiv
DO - 10.48550/arxiv.2609.19318
UR - https://arxiv.org/abs/2609.19318
AB - Clinical data abstraction, the process of distilling structured information from patient records, plays a key role in advancing knowledge about diseases such as cancer. Information extraction (IE) with large language models (LLMs) could accelerate this process, but it is unclear whether current frameworks effectively support clinical researchers without AI expertise. To address this, we co-designed an interactive LLM-based abstraction system called Libretto with seven cancer research teams, then evaluated the system's ability to help them answer real-world research questions. We found that while clinicians knew where and how to annotate complex concepts in patient notes, in twelve of fourteen tasks they faced barriers to replicating those intuitions with LLMs. Contextual note reliability judgments, difficulties in steering vibe-coded prompts, and inflexible evaluation strategies necessitated fundamental changes to the IE workflow. Our results highlight open problems for HCI research to bridge the gaps between AI data work tools and clinical users' needs.
ER -