Rethinking large language model evaluation in traditional Chinese medicine: toward quality-grounded standards and fine-grained assessment.
- 1WangQi Academy, National Institute of Traditional Chinese Medicine Constitution and Preventive Treatment of Diseases, Beijing University of Chinese Medicine, Beijing, China.
Abstract
Large language models (LLMs) are increasingly applied to clinical decision support for traditional Chinese medicine (TCM). However, current evaluation practices remain inadequate. Uncurated knowledge sources, such as un-selected ancient texts and web-crawled data, may contain outdated or prohibited recommendations. Specific clinical cases, even a single case, used as the benchmark may mislead the evaluation by individual experience instead of evidence-based consensus. Coarse-grained scoring strategies, such as true/false selections or holistic Likert ratings, obscure component-level errors and underestimate clinical complexity. TCM clinical reasoning is multidimensional, encompassing multiple coexisting syndromes and detailed procedural parameters. We propose a path forward built on two pillars: (1) structuring authoritative knowledge, including national clinical guidelines, industry standards, standard textbooks, the Chinese Pharmacopoeia, and consensus derived from large-scale high-quality cases; and (2) fine-grained, component-wise evaluation, in which complex TCM reasoning is decomposed into minimal assessable units across diagnosis and treatment, scored with component-level precision/recall and error grading, and judged primarily by the identification of critical errors that could cause medical harm, supplemented by expert-assessed global acceptability. Such quality-grounded, component-wise frameworks would enhance scientific credibility, support regulatory scrutiny and clinical adoption, and enable targeted model improvement aligned with the WHO Traditional Medicine Strategy 2025-2034.
Related
- Navigating intraoperative hypotension: rethinking traditional management for improved patient outcomes
- RoB2-Evaluator: Development and Technical Evaluation of a Rule-Constrained Large Language Model System for Cochrane Risk of Bias 2 Assessment
- Low-Intensity Continuous Ultrasound vs. Traditional Ultrasound: A Systematic Review and Descriptive Analysis on Past and Future Gold Standards for Deep-Tissue Diathermy in Clinical Medicine.
- Development and Evaluation of an Offline Vision-Language Model System for Estimating Nutrients from Meal Photographs on Personal Devices: The PLANS (Personal-device Local Assessment of Nutrients System) Study
- Evaluating different large language model performances in evidence-based dental implantology: a blinded controlled experimental evaluation using expert panel assessment.
BibTeX
@article{wang2026rethinking,
title = {{Rethinking large language model evaluation in traditional Chinese medicine: toward quality-grounded standards and fine-grained assessment.}},
author = {Huaiyu Wang and Ji Wang and Qi Wang},
journal = {Medical review (2021)},
year = {2026},
month = sep,
doi = {10.1515/mr-2026-0006},
url = {https://europepmc.org/article/MED/42923809},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Rethinking large language model evaluation in traditional Chinese medicine: toward quality-grounded standards and fine-grained assessment.
AU - Huaiyu Wang
AU - Ji Wang
AU - Qi Wang
PY - 2026
DA - 2026/09/01
JO - Medical review (2021)
DO - 10.1515/mr-2026-0006
UR - https://europepmc.org/article/MED/42923809
AB - Large language models (LLMs) are increasingly applied to clinical decision support for traditional Chinese medicine (TCM). However, current evaluation practices remain inadequate. Uncurated knowledge sources, such as un-selected ancient texts and web-crawled data, may contain outdated or prohibited recommendations. Specific clinical cases, even a single case, used as the benchmark may mislead the evaluation by individual experience instead of evidence-based consensus. Coarse-grained scoring strategies, such as true/false selections or holistic Likert ratings, obscure component-level errors and underestimate clinical complexity. TCM clinical reasoning is multidimensional, encompassing multiple coexisting syndromes and detailed procedural parameters. We propose a path forward built on two pillars: (1) structuring authoritative knowledge, including national clinical guidelines, industry standards, standard textbooks, the Chinese Pharmacopoeia, and consensus derived from large-scale high-quality cases; and (2) fine-grained, component-wise evaluation, in which complex TCM reasoning is decomposed into minimal assessable units across diagnosis and treatment, scored with component-level precision/recall and error grading, and judged primarily by the identification of critical errors that could cause medical harm, supplemented by expert-assessed global acceptability. Such quality-grounded, component-wise frameworks would enhance scientific credibility, support regulatory scrutiny and clinical adoption, and enable targeted model improvement aligned with the WHO Traditional Medicine Strategy 2025-2034.
SN - 2097-0773
SN - 2749-9642
ER -