An LLM-based motivational interviewing training system for Japanese clinical education
Abstract
Background: Motivational interviewing (MI) is an evidence-based approach for promoting behavior change in mental health, primary care, and public health. Effective MI training requires repeated practice with expert feedback, yet such resources remain scarce, and few tools support Japanese. Large language models (LLMs) offer scalable training, but whether LLM-based assessment agrees with expert evaluation remains largely unverified, particularly for Japanese.
Methods: We developed an LLM-based MI training system for Japanese clinical education and are validating its evaluation module against expert judgment. From book-derived case seeds, we generated 81 synthetic Japanese MI dialogues (27 scenarios x 3 intended counselor skill levels). The LLM evaluator assigns MITI 4.2.1 global scores (primary) and utterance-level behavior codes (secondary) with written rationales. Validation comprises (a) benchmarking against the official MITI 4.2.1 manual coding examples and (b) blinded coding of a held-out set by an expert rater, with a second expert reviewing uncertain judgments; agreement is assessed by intraclass correlation and weighted kappa.
Results: On 106 coding examples from the official MITI 4.2.1 manual, the evaluator achieved 71.7% accuracy (Cohen’s kappa=0.68), within the range of inter-coder agreement reported among trained human raters. Misclassifications concentrated in categories that also challenge human coders, such as simple versus complex reflections. Blinded expert coding of a held-out set is ongoing; agreement results will be presented at the conference.
Conclusions: Anchoring automated assessment to the official MITI standard and blinded expert judgment can make scalable MI training trustworthy for Japanese clinical education. Future work includes a feasibility pilot with medical students, strengthening workforce capacity for behavior change counseling.
Key messages: • An LLM-based system can provide scalable MI training for Japanese clinical education, addressing limited access to expert feedback and practice opportunities.
• Anchoring LLM evaluation to official MITI benchmarks and blinded expert coding provides a transparent template for validating AI-assisted skills training before educational use.
Related
- Motivational Interviewing for Pediatric Residents: A Case-Based Workshop on Adolescent Substance Use.
- Effectiveness of a Sequential Simulation and Online Learning Intervention on Psychiatric Mental Health Nurses’ Motivational Interviewing Knowledge, Confidence, and Skill Performance for Medication Adherence: A Cluster-randomized Controlled Trial
- Phase-sensitive integration of motivational interviewing and cognitive behavioral therapy for anxiety disorders: a scoping review.
- Effect of motivational interviews and peer education on adherence to pre-exposure prophylaxis among key populations in Kisangani: a randomized clinical trial.
- The Moderating Effects of Race and Ethnicity on a Motivational Interviewing Intervention for Improving Physical Health Among Patients With Peripheral Artery Disease: Secondary Analysis of a Pilot Randomized Trial
BibTeX
@article{arashi2026llmbased,
title = {{An LLM-based motivational interviewing training system for Japanese clinical education}},
author = {H Arashi and K Okumura and M Yasuda and N Otsuka and T Okada},
journal = {The European Journal of Public Health},
year = {2026},
month = oct,
doi = {10.1093/eurpub/ckag169.1220},
url = {https://doi.org/10.1093/eurpub/ckag169.1220},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - An LLM-based motivational interviewing training system for Japanese clinical education
AU - H Arashi
AU - K Okumura
AU - M Yasuda
AU - N Otsuka
AU - T Okada
PY - 2026
DA - 2026/10/01
JO - The European Journal of Public Health
DO - 10.1093/eurpub/ckag169.1220
UR - https://doi.org/10.1093/eurpub/ckag169.1220
AB - Background: Motivational interviewing (MI) is an evidence-based approach for promoting behavior change in mental health, primary care, and public health. Effective MI training requires repeated practice with expert feedback, yet such resources remain scarce, and few tools support Japanese. Large language models (LLMs) offer scalable training, but whether LLM-based assessment agrees with expert evaluation remains largely unverified, particularly for Japanese. Methods: We developed an LLM-based MI training system for Japanese clinical education and are validating its evaluation module against expert judgment. From book-derived case seeds, we generated 81 synthetic Japanese MI dialogues (27 scenarios x 3 intended counselor skill levels). The LLM evaluator assigns MITI 4.2.1 global scores (primary) and utterance-level behavior codes (secondary) with written rationales. Validation comprises (a) benchmarking against the official MITI 4.2.1 manual coding examples and (b) blinded coding of a held-out set by an expert rater, with a second expert reviewing uncertain judgments; agreement is assessed by intraclass correlation and weighted kappa. Results: On 106 coding examples from the official MITI 4.2.1 manual, the evaluator achieved 71.7% accuracy (Cohen’s kappa=0.68), within the range of inter-coder agreement reported among trained human raters. Misclassifications concentrated in categories that also challenge human coders, such as simple versus complex reflections. Blinded expert coding of a held-out set is ongoing; agreement results will be presented at the conference. Conclusions: Anchoring automated assessment to the official MITI standard and blinded expert judgment can make scalable MI training trustworthy for Japanese clinical education. Future work includes a feasibility pilot with medical students, strengthening workforce capacity for behavior change counseling. Key messages: • An LLM-based system can provide scalable MI training for Japanese clinical education, addressing limited access to expert feedback and practice opportunities. • Anchoring LLM evaluation to official MITI benchmarks and blinded expert coding provides a transparent template for validating AI-assisted skills training before educational use.
SN - 1101-1262
SN - 1464-360X
ER -