Evaluating Chinese large language models for HIV/AIDS health information: a multidimensional comparative study of response quality.
- 1Department of Biomedical Informatics, School of Life Sciences, Central South University, Changsha, Hunan, China.
- 2School of Medical Information Engineering, Jining Medical University, Rizhao, Shandong, China.
Abstract
Background
Chinese large language models (LLMs) are increasingly used for HIV/AIDS health information, yet their multidimensional performance remains unevaluated. This study systematically assessed five mainstream Chinese LLMs across accuracy, caring, completeness, understandability, and actionability.Methods
A cross-sectional evaluation was conducted using 50 standardized HIV/AIDS questions. Each model's responses were rated by three independent experts on five dimensions, including adapted PEMAT-based items. Latent profile analysis (LPA) and partial correlation network analysis were employed to identify quality typologies and interdimensional dependencies. Chi-square tests examined associations between profile membership and model identity or question domain.Results
The four general-purpose LLMs showed high accuracy (mean > 4.5/5) and completeness (86%-98% full marks), with no significant differences among them (all P-FDR > 0.05). HuatuoGPT-II, the only medical-domain-specific model, scored significantly lower on both dimensions (all P < 0.001). Its responses were less consistent with current guidelines and included one instance of implicit stigmatizing language. All models showed systemic deficits in caring (median = 3.0/5) and actionability (max = 1.70/3). Understandability showed a ceiling effect and no significant profile-level differences, although small model-level differences were observed. LPA identified four profiles: High Accuracy and Completeness (50.0%), Moderate-Balanced (3.6%), Low-Quality (18.8%), and High Completeness and Actionability-Oriented (27.6%). Network analysis identified completeness as the most central node (strength = 0.813) and actionability as a key bridge (betweenness = 0.167). A weak negative partial correlation between accuracy and actionability (r = -0.083) suggested a potential trade-off between technical precision and practical guidance at the statistical level. Profile distributions differed by model (χ² = 124.99, P < 0.001) but not domain (P = 0.625), indicating heterogeneity stems from model-specific characteristics. Yuanbao-Hy3 showed the most balanced humanistic-technical profile. DeepSeek-V4 excelled technically but lacked caring.Conclusions
In this evaluation, the four general-purpose models outperformed the single medical-domain-specific model examined in factual accuracy and completeness. However, systemic deficits in empathetic communication and actionable guidance persist. Model selection should prioritize holistic profile distributions over single-dimension scores. Developers must integrate de-stigmatizing corpora and action-oriented alignment; clinicians should position AI as supplementary to professional care. Future evaluations should adopt person-centred, multi-turn, and real-user validated frameworks.
Related
- A Multidimensional Evaluation of Large Language Model Responses to the 2024 ESC Hypertension Guidelines: A Comparative Study
- Evaluating Large Language Models as Tools to Navigate Researchers in Rapidly Evolving Research Landscapes: A Case Study in Cancer Drug Response Prediction
- Comparative performance of nine large language models on professional Parkinson’s disease tasks: a multidimensional benchmark study
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- Evaluating large language models in patient education: a comparative analysis addressing frequently asked questions in peri-acetabular osteotomy
BibTeX
@article{gu2026evaluating,
title = {{Evaluating Chinese large language models for HIV/AIDS health information: a multidimensional comparative study of response quality.}},
author = {Jinhao Gu and Jianwei Liu and Yunrong Liu and Pingping Wang and Dehua Hu and Qizhen Zhu},
journal = {Frontiers in public health},
year = {2026},
month = oct,
doi = {10.3389/fpubh.2026.1966522},
url = {https://europepmc.org/article/MED/42903209},
note = {CC BY},
}
RIS
TY - JOUR
TI - Evaluating Chinese large language models for HIV/AIDS health information: a multidimensional comparative study of response quality.
AU - Jinhao Gu
AU - Jianwei Liu
AU - Yunrong Liu
AU - Pingping Wang
AU - Dehua Hu
AU - Qizhen Zhu
PY - 2026
DA - 2026/10/06
JO - Frontiers in public health
DO - 10.3389/fpubh.2026.1966522
UR - https://europepmc.org/article/MED/42903209
AB - Background Chinese large language models (LLMs) are increasingly used for HIV/AIDS health information, yet their multidimensional performance remains unevaluated. This study systematically assessed five mainstream Chinese LLMs across accuracy, caring, completeness, understandability, and actionability.Methods A cross-sectional evaluation was conducted using 50 standardized HIV/AIDS questions. Each model's responses were rated by three independent experts on five dimensions, including adapted PEMAT-based items. Latent profile analysis (LPA) and partial correlation network analysis were employed to identify quality typologies and interdimensional dependencies. Chi-square tests examined associations between profile membership and model identity or question domain.Results The four general-purpose LLMs showed high accuracy (mean > 4.5/5) and completeness (86%-98% full marks), with no significant differences among them (all P-FDR > 0.05). HuatuoGPT-II, the only medical-domain-specific model, scored significantly lower on both dimensions (all P < 0.001). Its responses were less consistent with current guidelines and included one instance of implicit stigmatizing language. All models showed systemic deficits in caring (median = 3.0/5) and actionability (max = 1.70/3). Understandability showed a ceiling effect and no significant profile-level differences, although small model-level differences were observed. LPA identified four profiles: High Accuracy and Completeness (50.0%), Moderate-Balanced (3.6%), Low-Quality (18.8%), and High Completeness and Actionability-Oriented (27.6%). Network analysis identified completeness as the most central node (strength = 0.813) and actionability as a key bridge (betweenness = 0.167). A weak negative partial correlation between accuracy and actionability (r = -0.083) suggested a potential trade-off between technical precision and practical guidance at the statistical level. Profile distributions differed by model (χ² = 124.99, P < 0.001) but not domain (P = 0.625), indicating heterogeneity stems from model-specific characteristics. Yuanbao-Hy3 showed the most balanced humanistic-technical profile. DeepSeek-V4 excelled technically but lacked caring.Conclusions In this evaluation, the four general-purpose models outperformed the single medical-domain-specific model examined in factual accuracy and completeness. However, systemic deficits in empathetic communication and actionable guidance persist. Model selection should prioritize holistic profile distributions over single-dimension scores. Developers must integrate de-stigmatizing corpora and action-oriented alignment; clinicians should position AI as supplementary to professional care. Future evaluations should adopt person-centred, multi-turn, and real-user validated frameworks.
SN - 2296-2565
ER -