Large Language Model Chatbot Responses to Cancer Survivorship Questions in Hong Kong: Bilingual Evaluation and Prompt Optimization Study.
5 affiliations
- 1Department of Family Medicine and Primary Care, School of Clinical Medicine, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, SAR, China (Hong Kong).
- 2Independent researcher, Hong Kong, SAR, China (Hong Kong).
- 3Department of Clinical Oncology, Pamela Youde Nethersole Eastern Hospital, Hong Kong, SAR, China (Hong Kong).
- 4Division of Urology, Department of Surgery, School of Clinical Medicine, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, SAR, China (Hong Kong).
- 5Department of Family Medicine, University of Hong Kong - Shenzhen Hospital, Shenzhen, Guangdong, China.
Abstract
Background
As the population of survivors of cancer grows and new information technologies become widespread in Hong Kong, survivors of cancer increasingly seek ongoing care and support information from large language models (LLMs). While these tools provide immediate conversational responses, they carry substantial risks of generating inaccurate, unsafe, or generic medical advice. Evaluating LLM response performance is particularly critical in Hong Kong's bilingual health care context.Objective
This study aimed to evaluate the clinical utility, safety, understandability, and readability of 4 popular commercial LLMs answering cancer survivorship questions in English and Traditional Chinese and to explore whether a structured prompting strategy improves response performance.Methods
We conducted a 2-phase evaluation study combining quantitative scoring with qualitative classification of unsafe responses. In phase 1, 25 expert-curated questions spanning 5 survivorship domains were input into 4 LLMs: Google Gemini 3.1 Pro, HK Chat-0.6.2, DeepSeek-V3.2, and Kimi-K2.5. A Delphi expert panel evaluated the baseline responses for clinical utility using a 5-point scale and assessed safety via binary categorization. Understandability was measured using the Patient Education Materials Assessment Tool (PEMAT). Readability was assessed via the Flesch-Kincaid Grade Level for English and lexical richness for Traditional Chinese. In phase 2, experts developed an optimized context, role, audience, format, task, and tone (CRAFT) prompt, and the resulting LLM responses were compared with baseline outputs in an exploratory within-sample analysis using paired statistical tests.Results
Baseline evaluations identified Google Gemini 3.1 Pro as the most clinically useful model in English (mean 3.99/5, SD 0.16) and Traditional Chinese (mean 4.21/5, SD 0.20). Unconstrained models demonstrated language-dependent safety vulnerabilities: English errors (up to 24%) centered on service mismatches and definitive interpretations, whereas Traditional Chinese errors (up to 20%) involved prescriptive disease management and unverified adjunctive therapies. Within each language, the models producing the most readable text were not those achieving the highest clinical utility; this relationship was not formally tested. Applying the CRAFT prompt to Gemini 3.1 Pro significantly improved clinical utility across both languages (English: mean 4.68, SD 0.18, P<.001; Traditional Chinese: 4.66, SD 0.23, P<.001). After prompting, the 2 unsafe English baseline responses were reclassified as safe (English 23/25 to 25/25 safe; Traditional Chinese 25/25 at both time points). This transition was not statistically significant on a post hoc exact McNemar test (P=.50) and is reported as descriptive.Conclusions
LLMs exhibited language-divergent clinical safety vulnerabilities when answering cancer survivorship queries, and within each language, the most readable models were not the most clinically useful. In an exploratory within-sample comparison, expert-designed CRAFT prompting improved clinical utility and corrected the unsafe outputs seen at baseline without reducing understandability. Safe deployment of LLMs in multilingual oncology contexts requires strict validation combined with structured, role-restrictive constraints.
Related
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Comparative evaluation of ChatGPT and gemini responses to patient-oriented questions on breast cancer
- Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.
- Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy
- Comparative evaluation of four large language models for postoperative patient education after septorhinoplasty: A bilingual cross-sectional study
BibTeX
@article{an2026large,
title = {{Large Language Model Chatbot Responses to Cancer Survivorship Questions in Hong Kong: Bilingual Evaluation and Prompt Optimization Study.}},
author = {Haoyu An and Wanshu Huang and Rachel Tsoi Man Cheung and Qijun Du and Rong Na and David Ka Ki Wong},
journal = {Journal of medical Internet research},
year = {2026},
month = oct,
doi = {10.2196/105778},
url = {https://europepmc.org/article/MED/42849027},
note = {CC BY},
}
RIS
TY - JOUR
TI - Large Language Model Chatbot Responses to Cancer Survivorship Questions in Hong Kong: Bilingual Evaluation and Prompt Optimization Study.
AU - Haoyu An
AU - Wanshu Huang
AU - Rachel Tsoi Man Cheung
AU - Qijun Du
AU - Rong Na
AU - David Ka Ki Wong
PY - 2026
DA - 2026/10/08
JO - Journal of medical Internet research
DO - 10.2196/105778
UR - https://europepmc.org/article/MED/42849027
AB - Background As the population of survivors of cancer grows and new information technologies become widespread in Hong Kong, survivors of cancer increasingly seek ongoing care and support information from large language models (LLMs). While these tools provide immediate conversational responses, they carry substantial risks of generating inaccurate, unsafe, or generic medical advice. Evaluating LLM response performance is particularly critical in Hong Kong's bilingual health care context.Objective This study aimed to evaluate the clinical utility, safety, understandability, and readability of 4 popular commercial LLMs answering cancer survivorship questions in English and Traditional Chinese and to explore whether a structured prompting strategy improves response performance.Methods We conducted a 2-phase evaluation study combining quantitative scoring with qualitative classification of unsafe responses. In phase 1, 25 expert-curated questions spanning 5 survivorship domains were input into 4 LLMs: Google Gemini 3.1 Pro, HK Chat-0.6.2, DeepSeek-V3.2, and Kimi-K2.5. A Delphi expert panel evaluated the baseline responses for clinical utility using a 5-point scale and assessed safety via binary categorization. Understandability was measured using the Patient Education Materials Assessment Tool (PEMAT). Readability was assessed via the Flesch-Kincaid Grade Level for English and lexical richness for Traditional Chinese. In phase 2, experts developed an optimized context, role, audience, format, task, and tone (CRAFT) prompt, and the resulting LLM responses were compared with baseline outputs in an exploratory within-sample analysis using paired statistical tests.Results Baseline evaluations identified Google Gemini 3.1 Pro as the most clinically useful model in English (mean 3.99/5, SD 0.16) and Traditional Chinese (mean 4.21/5, SD 0.20). Unconstrained models demonstrated language-dependent safety vulnerabilities: English errors (up to 24%) centered on service mismatches and definitive interpretations, whereas Traditional Chinese errors (up to 20%) involved prescriptive disease management and unverified adjunctive therapies. Within each language, the models producing the most readable text were not those achieving the highest clinical utility; this relationship was not formally tested. Applying the CRAFT prompt to Gemini 3.1 Pro significantly improved clinical utility across both languages (English: mean 4.68, SD 0.18, P<.001; Traditional Chinese: 4.66, SD 0.23, P<.001). After prompting, the 2 unsafe English baseline responses were reclassified as safe (English 23/25 to 25/25 safe; Traditional Chinese 25/25 at both time points). This transition was not statistically significant on a post hoc exact McNemar test (P=.50) and is reported as descriptive.Conclusions LLMs exhibited language-divergent clinical safety vulnerabilities when answering cancer survivorship queries, and within each language, the most readable models were not the most clinically useful. In an exploratory within-sample comparison, expert-designed CRAFT prompting improved clinical utility and corrected the unsafe outputs seen at baseline without reducing understandability. Safe deployment of LLMs in multilingual oncology contexts requires strict validation combined with structured, role-restrictive constraints.
SN - 1439-4456
SN - 1438-8871
ER -