Multilingual evaluation of a large language model-based primary care chatbot.
- 1AI-4-AI Lab, University of Pennsylvania, 3700 Hamilton Walk, Philadelphia, PA, 19104, United States.
- 2Palliative and Advanced Illness Research (PAIR) Center, Perelman School of Medicine, University of Pennsylvania, 3600 Civic Center Boulevard 7th Floor East, Philadelphia, PA, 19104, United States.
Abstract
Objective
Large language model (LLM) chatbots show promise for pre-visit planning, but their English-centric development raises equity concerns for multilingual deployment. We evaluated the multilingual capabilities of the Primary Care Pre-visit Chatbot (PCP-Bot), a GPT-4o-based pre-visit chatbot, across English, Mandarin Chinese, Spanish, and Hindi.Materials and methods
In this mixed-methods study, 31 bilingual participants (11 Mandarin, 10 Spanish, and 10 Hindi) role-played as patients across 5 synthetic primary care cases, each first in English, then in their second language. Participants completed validated instruments (Bot Usability Scale [BUS-11], NASA Task Load Index [NASA-TLX], Human-Computer Trust Scale [HCTS], and Technology Acceptance Model-Perceived Ease of Use [TAM-PEU]), an adapted summary quality rubric (fluency, adequacy, meaning, and severity), and provided open-ended feedback.Results
English interactions scored higher than non-English interactions on usability (difference = 0.15), conversation quality (0.62), and summary quality (0.25; all P < .001). Usability and conversation quality gaps varied by language (both P < .01), driven by Spanish and Hindi; Mandarin Chinese showed neither (P = .087). Trust, workload, and ease-of-use remained consistent across languages. Participants described PCP-Bot as natural and accurate overall but reported repetition, transcription errors, and more frequent breakdowns in non-English interactions.Discussion
Multilingual capability in pre-visit workflows appears feasible for LLMs under controlled conditions in the 4 high-resource languages evaluated, with high absolute performance in each. Disparities were language-dependent: conversational performance declined most for Spanish and Hindi, while summary quality showed a small, uniform English advantage, suggesting addressable language-specific weaknesses rather than a fixed limitation of English-developed systems.Conclusion
Large language models can support multilingual pre-visit interactions, but performance varies by language. Per-language validation and further testing are needed prior to clinical rollout.
Related
- AFP assistant: a retrieval-augmented generation and large language model-powered multilingual polio chatbot for low-resource language communities
- Large Language Model Chatbot Responses to Cancer Survivorship Questions in Hong Kong: Bilingual Evaluation and Prompt Optimization Study.
- Expert-referenced evaluation of large language models for primary suspect drug identification in the FAERS database
- Utilization of a HIPAA-compliant large language model chatbot in an academic pediatric medical center
- Performance of Large Language Models in Image-Based Evaluation of Dupuytren Contracture
BibTeX
@article{chen2026multilingual,
title = {{Multilingual evaluation of a large language model-based primary care chatbot.}},
author = {Pei-Lun Chen and Amogh Ananda Rao and Sydney Pugh and Yingying Lu and Kevin B Johnson},
journal = {JAMIA open},
year = {2026},
month = oct,
doi = {10.1093/jamiaopen/ooag207},
url = {https://europepmc.org/article/MED/42921364},
note = {CC BY-NC},
}
RIS
TY - JOUR
TI - Multilingual evaluation of a large language model-based primary care chatbot.
AU - Pei-Lun Chen
AU - Amogh Ananda Rao
AU - Sydney Pugh
AU - Yingying Lu
AU - Kevin B Johnson
PY - 2026
DA - 2026/10/23
JO - JAMIA open
DO - 10.1093/jamiaopen/ooag207
UR - https://europepmc.org/article/MED/42921364
AB - Objective Large language model (LLM) chatbots show promise for pre-visit planning, but their English-centric development raises equity concerns for multilingual deployment. We evaluated the multilingual capabilities of the Primary Care Pre-visit Chatbot (PCP-Bot), a GPT-4o-based pre-visit chatbot, across English, Mandarin Chinese, Spanish, and Hindi.Materials and methods In this mixed-methods study, 31 bilingual participants (11 Mandarin, 10 Spanish, and 10 Hindi) role-played as patients across 5 synthetic primary care cases, each first in English, then in their second language. Participants completed validated instruments (Bot Usability Scale [BUS-11], NASA Task Load Index [NASA-TLX], Human-Computer Trust Scale [HCTS], and Technology Acceptance Model-Perceived Ease of Use [TAM-PEU]), an adapted summary quality rubric (fluency, adequacy, meaning, and severity), and provided open-ended feedback.Results English interactions scored higher than non-English interactions on usability (difference = 0.15), conversation quality (0.62), and summary quality (0.25; all P < .001). Usability and conversation quality gaps varied by language (both P < .01), driven by Spanish and Hindi; Mandarin Chinese showed neither (P = .087). Trust, workload, and ease-of-use remained consistent across languages. Participants described PCP-Bot as natural and accurate overall but reported repetition, transcription errors, and more frequent breakdowns in non-English interactions.Discussion Multilingual capability in pre-visit workflows appears feasible for LLMs under controlled conditions in the 4 high-resource languages evaluated, with high absolute performance in each. Disparities were language-dependent: conversational performance declined most for Spanish and Hindi, while summary quality showed a small, uniform English advantage, suggesting addressable language-specific weaknesses rather than a fixed limitation of English-developed systems.Conclusion Large language models can support multilingual pre-visit interactions, but performance varies by language. Per-language validation and further testing are needed prior to clinical rollout.
SN - 2574-2531
ER -