Can large language models extract travel history from clinical text? A multi-annotator benchmark study
- 1University of Oxford
Abstract
International travel can increase infection risk and spread antimicrobial resistance (AMR), yet clinical systems rarely capture travel history in a structured format. Extracting this information from free-text clinical documentation at scale could improve alerts for high-consequence infections and AMR surveillance. We developed an annotation framework covering key aspects of travel history, including destination, exposures, travel duration, and multiple temporal variables. The framework was used to annotate a corpus of 100 clinical case reports by five clinicians, yielding over 5,000 clinician annotations. Several LLMs were evaluated on the same extraction task, with their outputs compared against clinician consensus. Agreement among clinicians was high for straightforward variables such as presence of travel and destination, but dropped substantially for temporal variables such as time since return and symptom onset relative to return. LLMs showed a similar pattern. They performed well on high-level variables but struggled with the same temporal fields that challenged clinicians. Model errors were also concentrated in cases where clinicians disagreed. Across models, outputs were more sensitive to sampling temperature than to prompt wording. A smaller open-weight model performed similarly to substantially larger models on several variables but remained weaker on temporal fields. Overall, LLMs could extract some travel history variables, but reliable extraction of temporal information remained difficult.
Related
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- Extracting smoking history from clinical notes for lung cancer screening decision support: comparing a structured-judgment model with general-purpose large language models
- Development and expert evaluation of a culturally adapted large language model for Malay clinical history-taking simulation: mixed methods study
- Adjudicator sensitivity of safety estimates in large language models for open-ended cardiovascular reasoning: a blinded multi-adjudicator benchmark study running title: adjudicator variability in LLM safety
- PsyEval: a comprehensive large language model evaluation benchmark for mental health
BibTeX
@misc{kanithi2026large,
title = {{Can large language models extract travel history from clinical text? A multi-annotator benchmark study}},
author = {P. Kanithi and A. Pradhan and H. Cui and K. Johnstone and A. Luk and R. Hammond and C. Brown and A. S. Walker and D. W. Eyre},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.29.26363894},
url = {https://www.medrxiv.org/content/10.64898/2026.09.29.26363894v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Can large language models extract travel history from clinical text? A multi-annotator benchmark study
AU - P. Kanithi
AU - A. Pradhan
AU - H. Cui
AU - K. Johnstone
AU - A. Luk
AU - R. Hammond
AU - C. Brown
AU - A. S. Walker
AU - D. W. Eyre
PY - 2026
DA - 2026/09/29
PB - medRxiv
DO - 10.64898/2026.09.29.26363894
UR - https://www.medrxiv.org/content/10.64898/2026.09.29.26363894v1
AB - International travel can increase infection risk and spread antimicrobial resistance (AMR), yet clinical systems rarely capture travel history in a structured format. Extracting this information from free-text clinical documentation at scale could improve alerts for high-consequence infections and AMR surveillance. We developed an annotation framework covering key aspects of travel history, including destination, exposures, travel duration, and multiple temporal variables. The framework was used to annotate a corpus of 100 clinical case reports by five clinicians, yielding over 5,000 clinician annotations. Several LLMs were evaluated on the same extraction task, with their outputs compared against clinician consensus. Agreement among clinicians was high for straightforward variables such as presence of travel and destination, but dropped substantially for temporal variables such as time since return and symptom onset relative to return. LLMs showed a similar pattern. They performed well on high-level variables but struggled with the same temporal fields that challenged clinicians. Model errors were also concentrated in cases where clinicians disagreed. Across models, outputs were more sensitive to sampling temperature than to prompt wording. A smaller open-weight model performed similarly to substantially larger models on several variables but remained weaker on temporal fields. Overall, LLMs could extract some travel history variables, but reliable extraction of temporal information remained difficult.
ER -