A triaxial benchmark for assessing responses from large language models in traditional Chinese medicine
5 affiliations
- 1School of Artificial Intelligence in Traditional Chinese Medicine, Shanghai University of Traditional Chinese Medicine, Shanghai, China. [email protected]
- 2Shanghai Artificial Intelligence Laboratory, Shanghai, China
- 3Institute of Basic Theory for Chinese Medicine, China Academy of Chinese Medical Sciences, Beijing, China. [email protected]
- 4School of Artificial Intelligence in Traditional Chinese Medicine, Shanghai University of Traditional Chinese Medicine, Shanghai, China. [email protected]
- 5Shanghai Artificial Intelligence Laboratory, Shanghai, China. [email protected]
Abstract
Large language models (LLMs) have demonstrated exceptional capabilities across diverse NLP tasks and domains, including modern medicine. However, the systematic evaluation of LLM in traditional Chinese medicine (TCM) - a field with rich historical depth and clinical complexity -remains underexplored. To address this gap, we introduce TCM-3CEval, a comprehensive benchmark designed to assess LLMs in TCM across three critical dimensions: (1) Mastery of core TCM knowledge, (2) Understanding of classical TCM texts, and (3) Clinical decision-making. We conduct rigorous evaluations on diverse model categories, including general-purpose international models (e.g., GPT-4o), Chinese general models (e.g., InternLM), and medical domain-specific models (e.g., PLUSE). Our findings reveal a clear performance hierarchy: (i) Systemic limitations persist –all models exhibit pronounced deficiencies in specialized subdomains such as Meridian & Acupoint theory and Various TCM Schools, revealing critical gaps between current capabilities and clinical requirements. (ii) Cultural-contextual alignment proves essential – models trained with Chinese linguistic and cultural priors significantly outperform international counterparts in classical text interpretation and clinical reasoning. TCM-3CEval establishes a standardized evaluation paradigm for AI in TCM, providing insights for optimizing LLMs in culturally grounded medical domains. The benchmark has been simultaneously uploaded to the recently launched Traditional Chinese Medicine (TCM) specialized track "In-depth Challenge for Comprehensive TCM Abilities: Fundamental Theories, Classical Interpretation, and Clinical Decision-making" on Medbench. This competition aims to comprehensively and multi-dimensionally assess LLMs’ capabilities in TCM, focusing on three core dimensions: mastery of basic TCM knowledge, understanding of classic texts, and clinical diagnosis and treatment decision-making. It is meticulously designed with multidimensional questions and real clinical case scenarios to accurately evaluate the professional TCM proficiency and practical application abilities of the models.
Related
- From examinations to consultations: a physician-reviewed Japanese benchmark for large language models in traditional Kampo medicine
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- Adjudicator sensitivity of safety estimates in large language models for open-ended cardiovascular reasoning: a blinded multi-adjudicator benchmark study running title: adjudicator variability in LLM safety
- The hospital cost of primary reverse total shoulder arthroplasty performed with traditional instrumentation: a Time-Driven Activity-Based Costing (TDABC) benchmark analysis in the era of digital enabling technology
- The Language of Trauma: A Comparative Analysis Using Natural Language Processing and Traditional Text Analysis Methods
BibTeX
@article{huang2026triaxial,
title = {{A triaxial benchmark for assessing responses from large language models in traditional Chinese medicine}},
author = {Tianai Huang and Lu Lu and Jiayuan Chen and Lihao Liu and Junjun He and Yuping Zhao and Wenchao Tang and Jie Xu},
journal = {Communications Medicine},
year = {2026},
month = may,
doi = {10.1038/s43856-026-01631-5},
url = {https://doi.org/10.1038/s43856-026-01631-5},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - A triaxial benchmark for assessing responses from large language models in traditional Chinese medicine
AU - Tianai Huang
AU - Lu Lu
AU - Jiayuan Chen
AU - Lihao Liu
AU - Junjun He
AU - Yuping Zhao
AU - Wenchao Tang
AU - Jie Xu
PY - 2026
DA - 2026/05/12
JO - Communications Medicine
DO - 10.1038/s43856-026-01631-5
UR - https://doi.org/10.1038/s43856-026-01631-5
AB - Large language models (LLMs) have demonstrated exceptional capabilities across diverse NLP tasks and domains, including modern medicine. However, the systematic evaluation of LLM in traditional Chinese medicine (TCM) - a field with rich historical depth and clinical complexity -remains underexplored. To address this gap, we introduce TCM-3CEval, a comprehensive benchmark designed to assess LLMs in TCM across three critical dimensions: (1) Mastery of core TCM knowledge, (2) Understanding of classical TCM texts, and (3) Clinical decision-making. We conduct rigorous evaluations on diverse model categories, including general-purpose international models (e.g., GPT-4o), Chinese general models (e.g., InternLM), and medical domain-specific models (e.g., PLUSE). Our findings reveal a clear performance hierarchy: (i) Systemic limitations persist –all models exhibit pronounced deficiencies in specialized subdomains such as Meridian & Acupoint theory and Various TCM Schools, revealing critical gaps between current capabilities and clinical requirements. (ii) Cultural-contextual alignment proves essential – models trained with Chinese linguistic and cultural priors significantly outperform international counterparts in classical text interpretation and clinical reasoning. TCM-3CEval establishes a standardized evaluation paradigm for AI in TCM, providing insights for optimizing LLMs in culturally grounded medical domains. The benchmark has been simultaneously uploaded to the recently launched Traditional Chinese Medicine (TCM) specialized track "In-depth Challenge for Comprehensive TCM Abilities: Fundamental Theories, Classical Interpretation, and Clinical Decision-making" on Medbench. This competition aims to comprehensively and multi-dimensionally assess LLMs’ capabilities in TCM, focusing on three core dimensions: mastery of basic TCM knowledge, understanding of classic texts, and clinical diagnosis and treatment decision-making. It is meticulously designed with multidimensional questions and real clinical case scenarios to accurately evaluate the professional TCM proficiency and practical application abilities of the models.
SN - 2730-664X
ER -