From examinations to consultations: a physician-reviewed Japanese benchmark for large language models in traditional Kampo medicine
- 1Department of Kampo Medicine, Tokai University School of Medicine
Abstract
ObjectivesPatients and clinicians increasingly ask large language models (LLMs) about Kampo, Japans traditional medicine. Yet evaluations of LLMs have measured accuracy on examination questions, which ask what a model knows; a consultation asks whether it can apply that knowledge to the patient at hand. No benchmark had measured that.
Materials and MethodsKampoBench comprises 74 consultation scenarios and 310 conversation-specific rubric criteria, following the design of HealthBench. Scenarios, criteria and reference answers were drafted by an LLM and revised by board-certified Kampo physicians. Ten current models from three vendors were evaluated. Because LLM graders are known to favor their own family, each response was graded by a model from each vendor.
ResultsThe best model reached 74.8-86.3% across graders, the physician-revised reference answers 87.9-92.4%. Rankings were stable across graders (Spearmans rho 0.952-0.976), and the family-favoring deviation was 0.4 to 0.6 percentage points, below the bootstrap standard deviation of one measurement; grader leniency varied by 14.3 points. Models were most reliable on criteria about advising that care be sought (83.6%) and least reliable on those requiring a known crude-drug risk to be tied to the case (17.5%), though 79.2% of those responses named the drug.
DiscussionThe shortfall lay in applying knowledge to the case rather than in holding it -- a distinction examination-format evaluations cannot make.
ConclusionEvaluation in traditional medicine should measure the judgments a consultation requires, not recall alone. Quoted alongside the grader that produced it, a KampoBench score is a usable measure of those judgments.
Lay summaryPeople increasingly ask artificial intelligence chatbots about Kampo, the traditional herbal medicine used alongside conventional care in Japan. Earlier tests used examination questions, which check what a chatbot knows. A real consultation asks something harder: whether the chatbot can apply that knowledge to the person asking. We built KampoBench, 74 realistic consultations in Japanese. For each one, specialist Kampo physicians reviewed a list of what a good answer should and should not contain, 310 items in all. We then tested ten current chatbots from three companies. The answers were scored by chatbots, and a chatbot might favor its own companys chatbots, so every answer was scored by one chatbot from each company. No such favoritism was detected, but the scorers differed in strictness. The best chatbot scored 75-86%, depending on the scorer; answers revised by physicians scored 88-92%. The chatbots were good at telling people to seek medical care. They were poor at relating a known risk of an herbal ingredient to the persons situation, such as noticing that two medicines the person already takes contain the same ingredient, often even when they named it. Tests of chatbots in traditional medicine should measure such judgments, not knowledge alone.
Related
- The hospital cost of primary reverse total shoulder arthroplasty performed with traditional instrumentation: a Time-Driven Activity-Based Costing (TDABC) benchmark analysis in the era of digital enabling technology
- Who Publishes During Residency? A Systematic Review and Meta-Analysis of Peer-Reviewed Publication Among Residents in the United States
- Prospective associations between psychosocial work exposures and general practitioner consultations in France
- Can LLMs Succeed in National Dental Examinations? A Comprehensive Analysis of Performance, Strengths, and Limitations
- Vaccines, Bias, and the Perils of Non-peer-reviewed Studies: Fueling Misinformation and Vaccine Hesitancy
BibTeX
@misc{takata2026from,
title = {{From examinations to consultations: a physician-reviewed Japanese benchmark for large language models in traditional Kampo medicine}},
author = {H. Takata and Y. Harada and H. Inoue and K. Uneda and T. Yoshino and T. Nogami},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.07.26365005},
url = {https://www.medrxiv.org/content/10.64898/2026.10.07.26365005v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - From examinations to consultations: a physician-reviewed Japanese benchmark for large language models in traditional Kampo medicine
AU - H. Takata
AU - Y. Harada
AU - H. Inoue
AU - K. Uneda
AU - T. Yoshino
AU - T. Nogami
PY - 2026
DA - 2026/10/07
PB - medRxiv
DO - 10.64898/2026.10.07.26365005
UR - https://www.medrxiv.org/content/10.64898/2026.10.07.26365005v1
AB - ObjectivesPatients and clinicians increasingly ask large language models (LLMs) about Kampo, Japans traditional medicine. Yet evaluations of LLMs have measured accuracy on examination questions, which ask what a model knows; a consultation asks whether it can apply that knowledge to the patient at hand. No benchmark had measured that. Materials and MethodsKampoBench comprises 74 consultation scenarios and 310 conversation-specific rubric criteria, following the design of HealthBench. Scenarios, criteria and reference answers were drafted by an LLM and revised by board-certified Kampo physicians. Ten current models from three vendors were evaluated. Because LLM graders are known to favor their own family, each response was graded by a model from each vendor. ResultsThe best model reached 74.8-86.3% across graders, the physician-revised reference answers 87.9-92.4%. Rankings were stable across graders (Spearmans rho 0.952-0.976), and the family-favoring deviation was 0.4 to 0.6 percentage points, below the bootstrap standard deviation of one measurement; grader leniency varied by 14.3 points. Models were most reliable on criteria about advising that care be sought (83.6%) and least reliable on those requiring a known crude-drug risk to be tied to the case (17.5%), though 79.2% of those responses named the drug. DiscussionThe shortfall lay in applying knowledge to the case rather than in holding it -- a distinction examination-format evaluations cannot make. ConclusionEvaluation in traditional medicine should measure the judgments a consultation requires, not recall alone. Quoted alongside the grader that produced it, a KampoBench score is a usable measure of those judgments. Lay summaryPeople increasingly ask artificial intelligence chatbots about Kampo, the traditional herbal medicine used alongside conventional care in Japan. Earlier tests used examination questions, which check what a chatbot knows. A real consultation asks something harder: whether the chatbot can apply that knowledge to the person asking. We built KampoBench, 74 realistic consultations in Japanese. For each one, specialist Kampo physicians reviewed a list of what a good answer should and should not contain, 310 items in all. We then tested ten current chatbots from three companies. The answers were scored by chatbots, and a chatbot might favor its own companys chatbots, so every answer was scored by one chatbot from each company. No such favoritism was detected, but the scorers differed in strictness. The best chatbot scored 75-86%, depending on the scorer; answers revised by physicians scored 88-92%. The chatbots were good at telling people to seek medical care. They were poor at relating a known risk of an herbal ingredient to the persons situation, such as noticing that two medicines the person already takes contain the same ingredient, often even when they named it. Tests of chatbots in traditional medicine should measure such judgments, not knowledge alone.
ER -