A cross-sectional comparison of five large language models in answering Chinese-language patient queries on visual field defects associated with brain diseases
4 affiliations
- 1The Second School of Clinical Medicine, Southern Medical University, Guangzhou, China
- 2School of Medical Technology and Engineering, Fujian Medical University, Fuzhou, China
- 3Department of Ophthalmology, Quzhou People’s Hospital, The Quzhou Affiliated Hospital, Wenzhou Medical University, Quzhou, China
- 4Shenzhen Eye Hospital, Shenzhen Eye Medical Center, Southern Medical University, Shenzhen, China
Abstract
BackgroundVisual field defects associated with brain diseases may arise from stroke, intracranial space-occupying lesions, and other central visual pathway disorders. Patient questions often span symptom recognition, explanation of causes, care triage, treatment, and rehabilitation. Large language models may improve access to patient education. However, their accuracy, logic, coherence, safety, and content accessibility have not been compared under controlled conditions in this high-risk, interdisciplinary Chinese-language setting. This study aimed to compare the performance of five selected large language models across multiple dimensions.MethodsIn this cross-sectional study, 30 fixed Chinese-language questions, refined by experts and phrased from a patient perspective, were submitted to ChatGPT 5.6 Sol, DeepSeek V4 Flash, Gemini 3.7 Flash, Qwen 3.8 Max, and Kimi K3 through a unified application programming interface (API) client. Each model generated one response per question in an independent single-turn session, with web retrieval disabled and conversation history cleared. Three raters, blinded to model identity, independently scored 150 randomly presented responses from 1 to 5 for accuracy, logic, coherence, safety, and content accessibility. Response time, word count, non-space character count, sentence count, paragraph count, and table count were also recorded. Models were compared using the Friedman test followed by Bonferroni-adjusted Wilcoxon signed-rank tests. Inter-rater agreement was assessed using Kendall’s W. Correlations were assessed using Spearman analysis with Benjamini-Hochberg false discovery rate correction.ResultsAgreement among the three raters was high across all five dimensions (Kendall’s W = 0.820–0.844; all P < 0.001). The five models differed in accuracy, coherence, and content accessibility (P < 0.001, P < 0.001, and P = 0.001, respectively). No statistically significant between-model differences were observed in logic or safety (P = 0.247 and P = 0.061, respectively). ChatGPT 5.6 Sol had the highest median accuracy score [4.67 (interquartile range, 4.00–5.00)], and Kimi K3 had the highest median coherence score [4.50 (4.00–5.00)]. Both models had a median content accessibility score of 4.67 (4.00–5.00). Response time, response length, and text structure also differed among models. However, their correlations with expert quality scores were generally weak (|ρ| ≤ 0.279).ConclusionThe five large language models exhibited distinctly multidimensional performance when answering Chinese-language patient questions about visual field defects associated with brain diseases. No single model consistently performed best across all clinically relevant dimensions. These findings are specific to the model versions tested on 15 August 2026; stability across repeated generations was not assessed. Response speed, length, and structure cannot substitute for direct assessment of medical quality. The absence of a statistically significant difference in safety ratings under controlled conditions does not imply equivalent or adequate clinical safety. Large language models are better suited to supporting patient education and clinician-patient communication under clinical supervision than to making diagnostic or treatment decisions independently.
Related
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
- Factors associated with the quality of urban geriatric healthcare services empowered by large language models and improvement pathways: a theoretical framework and cross-sectional study
- Evidence-based clinical question answering using retrieval-augmented generation (RAG) and open-source large language models
- Performance of large language models in answering orthodontic questions: A scoping review.
- Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study
BibTeX
@article{li2026crosssectional,
title = {{A cross-sectional comparison of five large language models in answering Chinese-language patient queries on visual field defects associated with brain diseases}},
author = {Shenghe Li and Ling Wang and Xiaoyan Wang and Ligang Jiang and Weihua Yang and Lu Xie},
journal = {Frontiers in Physiology},
year = {2026},
month = oct,
doi = {10.3389/fphys.2026.1993774},
url = {https://doi.org/10.3389/fphys.2026.1993774},
note = {CC BY},
}
RIS
TY - JOUR
TI - A cross-sectional comparison of five large language models in answering Chinese-language patient queries on visual field defects associated with brain diseases
AU - Shenghe Li
AU - Ling Wang
AU - Xiaoyan Wang
AU - Ligang Jiang
AU - Weihua Yang
AU - Lu Xie
PY - 2026
DA - 2026/10/23
JO - Frontiers in Physiology
DO - 10.3389/fphys.2026.1993774
UR - https://doi.org/10.3389/fphys.2026.1993774
AB - BackgroundVisual field defects associated with brain diseases may arise from stroke, intracranial space-occupying lesions, and other central visual pathway disorders. Patient questions often span symptom recognition, explanation of causes, care triage, treatment, and rehabilitation. Large language models may improve access to patient education. However, their accuracy, logic, coherence, safety, and content accessibility have not been compared under controlled conditions in this high-risk, interdisciplinary Chinese-language setting. This study aimed to compare the performance of five selected large language models across multiple dimensions.MethodsIn this cross-sectional study, 30 fixed Chinese-language questions, refined by experts and phrased from a patient perspective, were submitted to ChatGPT 5.6 Sol, DeepSeek V4 Flash, Gemini 3.7 Flash, Qwen 3.8 Max, and Kimi K3 through a unified application programming interface (API) client. Each model generated one response per question in an independent single-turn session, with web retrieval disabled and conversation history cleared. Three raters, blinded to model identity, independently scored 150 randomly presented responses from 1 to 5 for accuracy, logic, coherence, safety, and content accessibility. Response time, word count, non-space character count, sentence count, paragraph count, and table count were also recorded. Models were compared using the Friedman test followed by Bonferroni-adjusted Wilcoxon signed-rank tests. Inter-rater agreement was assessed using Kendall’s W. Correlations were assessed using Spearman analysis with Benjamini-Hochberg false discovery rate correction.ResultsAgreement among the three raters was high across all five dimensions (Kendall’s W = 0.820–0.844; all P < 0.001). The five models differed in accuracy, coherence, and content accessibility (P < 0.001, P < 0.001, and P = 0.001, respectively). No statistically significant between-model differences were observed in logic or safety (P = 0.247 and P = 0.061, respectively). ChatGPT 5.6 Sol had the highest median accuracy score [4.67 (interquartile range, 4.00–5.00)], and Kimi K3 had the highest median coherence score [4.50 (4.00–5.00)]. Both models had a median content accessibility score of 4.67 (4.00–5.00). Response time, response length, and text structure also differed among models. However, their correlations with expert quality scores were generally weak (|ρ| ≤ 0.279).ConclusionThe five large language models exhibited distinctly multidimensional performance when answering Chinese-language patient questions about visual field defects associated with brain diseases. No single model consistently performed best across all clinically relevant dimensions. These findings are specific to the model versions tested on 15 August 2026; stability across repeated generations was not assessed. Response speed, length, and structure cannot substitute for direct assessment of medical quality. The absence of a statistically significant difference in safety ratings under controlled conditions does not imply equivalent or adequate clinical safety. Large language models are better suited to supporting patient education and clinician-patient communication under clinical supervision than to making diagnostic or treatment decisions independently.
SN - 1664-042X
ER -