Cross-linguistic differences in responses to patient queries on osteonecrosis of the femoral head between English and Chinese-oriented large language models: a semantically matched bilingual cross-sectional study
- 1Department of Tuina, First Teaching Hospital of Tianjin University of Traditional Chinese Medicine
- 2National Clinical Research Center for Chinese Medicine
- 3The Institute of Traditional Chinese Medicine Massage of Tianjin Health Commission
Abstract
Background: Large language models (LLMs) are capable of providing health information and consultation to patients with osteonecrosis of the femoral head (ONFH). However, owing to imbalances in the linguistic distribution of their training corpora, the quality, readability, and safety profiles of outputs generated by the same model may vary across language settings.
Objective: This study aimed to compare bilingual responses to ONFH patient queries generated by six state-of-the-art flagship LLMs, comprising three English-oriented models (ChatGPT-5.6-Sol, Gemini-3.1-Pro, and Claude-Fable-5) and three Chinese-oriented models (DeepSeek-V4-Pro, Qwen3.7-Max, and Doubao-Seed-2.1-Pro), and to multidimensionally evaluate their cross-linguistic capabilities for generating medical information in terms of information quality, actionability, clinical accuracy, safety risk, hallucination, and readability.
Methods: A question set comprising 36 semantically distinct patient queries regarding ONFH was developed, yielding 72 semantically matched English–Chinese paired prompts. Six state-of-the-art flagship LLMs were evaluated. Information quality was assessed using the Global Quality Score (GQS), DISCERN, the Quality Assessment of Medical Artificial Intelligence (QAMAI), and the 20-item Ensuring Quality Information for Patients (EQIP-20). Actionability was assessed using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P), while clinical accuracy was evaluated using a 5-point Likert scale. Safety risk was assessed using a graded classification framework, and prespecified criteria were established for identifying hallucinations. Readability was evaluated using language-specific readability metrics for English and Chinese responses. Statistical analyses included the Friedman test, paired Wilcoxon signed-rank test, and Spearman correlation analysis.
Results: A total of 432 responses generated by LLMs were included in this study, and good-to-excellent inter-rater reliability was observed for all evaluation indicators. Significant between-model differences in actionability and clinical accuracy were identified among the six models under both English and Chinese response conditions (all p < 0.05). Furthermore, clinical accuracy exhibited high cross-linguistic stability. ChatGPT-5.6-Sol showed high actionability and a relatively low frequency of Level 1 minor safety-related omissions under the English-language condition, whereas switching to Chinese was accompanied by reduced actionability and an increased frequency of Level 1 omissions. All observed nonzero safety events were Level 1 minor omissions; no Level 2 or Level 3 unsafe responses were identified in either language condition. Responses generated by all six models in both languages failed to reach the prespecified readability threshold for patient education.
Conclusion: The six LLMs showed significant inter-model differences in the actionability and clinical accuracy of responses to ONFH-related patient health queries in both English and Chinese settings, while clinical accuracy demonstrated favorable cross-linguistic stability. Notably, none of the six-model outputs met the applicable readability threshold for patient-targeted health education under either language condition. In clinical practice, model selection, prompting-language effects and hallucination-related risks should be comprehensively considered when deploying LLMs to deliver health information for patients with ONFH. At the present stage, LLM-generated content intended for direct patient-facing education requires cautious evaluation and human oversight.
Related
- Large Language Model Chatbot Responses to Cancer Survivorship Questions in Hong Kong: Bilingual Evaluation and Prompt Optimization Study.
- Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk.
- Delusion-like content emerges from common-cause inference over semantically disparate observations
- The effect of differences in viewing environment and method on physiological responses and perceptual stability while viewing three-dimensional images
- Daytime illumination shapes sex differences in sleep and circadian responses to nighttime light
BibTeX
@article{cao2026crosslinguistic,
title = {{Cross-linguistic differences in responses to patient queries on osteonecrosis of the femoral head between English and Chinese-oriented large language models: a semantically matched bilingual cross-sectional study}},
author = {ShiHang Cao and JiaChen Liang and HaiNing Zhang and XueRui Cui and Qi Zhao and YuanXu Cheng and YaDi Ren and Yu Jiang and HuaNan Li},
journal = {Frontiers in Public Health},
year = {2026},
month = oct,
doi = {10.3389/fpubh.2026.1985180},
url = {https://www.frontiersin.org/articles/10.3389/fpubh.2026.1985180/full},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Cross-linguistic differences in responses to patient queries on osteonecrosis of the femoral head between English and Chinese-oriented large language models: a semantically matched bilingual cross-sectional study
AU - ShiHang Cao
AU - JiaChen Liang
AU - HaiNing Zhang
AU - XueRui Cui
AU - Qi Zhao
AU - YuanXu Cheng
AU - YaDi Ren
AU - Yu Jiang
AU - HuaNan Li
PY - 2026
DA - 2026/10/28
JO - Frontiers in Public Health
DO - 10.3389/fpubh.2026.1985180
UR - https://www.frontiersin.org/articles/10.3389/fpubh.2026.1985180/full
AB - Background: Large language models (LLMs) are capable of providing health information and consultation to patients with osteonecrosis of the femoral head (ONFH). However, owing to imbalances in the linguistic distribution of their training corpora, the quality, readability, and safety profiles of outputs generated by the same model may vary across language settings. Objective: This study aimed to compare bilingual responses to ONFH patient queries generated by six state-of-the-art flagship LLMs, comprising three English-oriented models (ChatGPT-5.6-Sol, Gemini-3.1-Pro, and Claude-Fable-5) and three Chinese-oriented models (DeepSeek-V4-Pro, Qwen3.7-Max, and Doubao-Seed-2.1-Pro), and to multidimensionally evaluate their cross-linguistic capabilities for generating medical information in terms of information quality, actionability, clinical accuracy, safety risk, hallucination, and readability. Methods: A question set comprising 36 semantically distinct patient queries regarding ONFH was developed, yielding 72 semantically matched English–Chinese paired prompts. Six state-of-the-art flagship LLMs were evaluated. Information quality was assessed using the Global Quality Score (GQS), DISCERN, the Quality Assessment of Medical Artificial Intelligence (QAMAI), and the 20-item Ensuring Quality Information for Patients (EQIP-20). Actionability was assessed using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P), while clinical accuracy was evaluated using a 5-point Likert scale. Safety risk was assessed using a graded classification framework, and prespecified criteria were established for identifying hallucinations. Readability was evaluated using language-specific readability metrics for English and Chinese responses. Statistical analyses included the Friedman test, paired Wilcoxon signed-rank test, and Spearman correlation analysis. Results: A total of 432 responses generated by LLMs were included in this study, and good-to-excellent inter-rater reliability was observed for all evaluation indicators. Significant between-model differences in actionability and clinical accuracy were identified among the six models under both English and Chinese response conditions (all p < 0.05). Furthermore, clinical accuracy exhibited high cross-linguistic stability. ChatGPT-5.6-Sol showed high actionability and a relatively low frequency of Level 1 minor safety-related omissions under the English-language condition, whereas switching to Chinese was accompanied by reduced actionability and an increased frequency of Level 1 omissions. All observed nonzero safety events were Level 1 minor omissions; no Level 2 or Level 3 unsafe responses were identified in either language condition. Responses generated by all six models in both languages failed to reach the prespecified readability threshold for patient education. Conclusion: The six LLMs showed significant inter-model differences in the actionability and clinical accuracy of responses to ONFH-related patient health queries in both English and Chinese settings, while clinical accuracy demonstrated favorable cross-linguistic stability. Notably, none of the six-model outputs met the applicable readability threshold for patient-targeted health education under either language condition. In clinical practice, model selection, prompting-language effects and hallucination-related risks should be comprehensively considered when deploying LLMs to deliver health information for patients with ONFH. At the present stage, LLM-generated content intended for direct patient-facing education requires cautious evaluation and human oversight.
SN - 2296-2565
ER -