Using Vision and Large Language Models for Psychological Intervention Competency Assessment: A Pilot Study of Agreement with Clinician Ratings
- 1Nationwide Children's Hospital
Abstract
Introduction: Supervision in psychological intervention training is limited by the time needed to evaluate simulated sessions. We tested a Vision Language Model (VLM) and Large Language Model (LLM) pipeline that assesses competency from verbal and nonverbal communication. Method: Gemini-3.1-Pro was used to annotate behaviors in 25 simulated therapy videos and incorporate them into transcripts, and ChatGPT-6.1-Sol rated them against the ENACT scale. Two clinicians verified VLM-identified behaviors and four independently rated each video. Agreement was estimated with Gwet's AC1 and ICC. Results: Of 1,558 VLM-identified behaviors, 70% were confirmed by both clinicians and 93% by at least one. AI-clinician agreement on therapeutic behavior attributes (AC1=0.73, 95% CI 0.70-0.76) approached between-clinician agreement (0.75, 0.72-0.78). AI-clinician ICCs on therapeutic domain levels (median 0.57) were modestly lower than between-clinician ICCs (0.67). Discussion: The pipeline agreed with clinicians nearly as well as clinicians agreed with each other and shows promise as a supplementary evaluator in therapist training.
Related
- 48-Week Real-World Atopic Dermatitis Study on Minimal Disease Activity, Associated Factors, and Agreement Between Clinician- and Patient-Reported Outcomes
- Assessment in competency-based medical education for anesthesiology: validity evidence, equity, and faculty readiness
- The quest for an ideal EEG competency assessment
- Optimizing the Patient Experience Through Clinician Coaching or Communication Intervention: A Randomized Pilot Study
- Remote Video Assessment of Ataxia via SARAhome: Correlation of Pose‐Derived Endpoints with Clinician Severity Ratings
BibTeX
@misc{liu2026using,
title = {{Using Vision and Large Language Models for Psychological Intervention Competency Assessment: A Pilot Study of Agreement with Clinician Ratings}},
author = {Y. Liu and W. Leever and M. Stanco and Z. Shook and P. Addico and S. Gleeson and F. Nugroho and G. Varney and E. Youngstrom and E. Sezgin},
howpublished = {medRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.02.26367108},
url = {https://www.medrxiv.org/content/10.64898/2026.11.02.26367108v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Using Vision and Large Language Models for Psychological Intervention Competency Assessment: A Pilot Study of Agreement with Clinician Ratings
AU - Y. Liu
AU - W. Leever
AU - M. Stanco
AU - Z. Shook
AU - P. Addico
AU - S. Gleeson
AU - F. Nugroho
AU - G. Varney
AU - E. Youngstrom
AU - E. Sezgin
PY - 2026
DA - 2026/11/02
PB - medRxiv
DO - 10.64898/2026.11.02.26367108
UR - https://www.medrxiv.org/content/10.64898/2026.11.02.26367108v1
AB - Introduction: Supervision in psychological intervention training is limited by the time needed to evaluate simulated sessions. We tested a Vision Language Model (VLM) and Large Language Model (LLM) pipeline that assesses competency from verbal and nonverbal communication. Method: Gemini-3.1-Pro was used to annotate behaviors in 25 simulated therapy videos and incorporate them into transcripts, and ChatGPT-6.1-Sol rated them against the ENACT scale. Two clinicians verified VLM-identified behaviors and four independently rated each video. Agreement was estimated with Gwet's AC1 and ICC. Results: Of 1,558 VLM-identified behaviors, 70% were confirmed by both clinicians and 93% by at least one. AI-clinician agreement on therapeutic behavior attributes (AC1=0.73, 95% CI 0.70-0.76) approached between-clinician agreement (0.75, 0.72-0.78). AI-clinician ICCs on therapeutic domain levels (median 0.57) were modestly lower than between-clinician ICCs (0.67). Discussion: The pipeline agreed with clinicians nearly as well as clinicians agreed with each other and shows promise as a supplementary evaluator in therapist training.
ER -