Feasibility and Measurement Properties of a Large Language Model-Based Virtual Patient Laboratory for Clinical Reasoning Training in Physical Therapy Students: Single-Arm Pilot Study
- 1Department of Physical Therapy, Occupational Therapy, Rehabilitation and Physical Medicine, Rey Juan Carlos University, Alcorcon, Madrid, Spain
Abstract
BackgroundLarge language models (LLMs) configured as virtual patients may expand structured practice of clinical reasoning in health professions education. Before testing effectiveness, it must be established whether students engage with them and whether the resulting scores are complete, reliable, and related to performance with real patients. Our earlier pilot randomized trial found very low engagement with a less structured LLM-based approach.
ObjectiveThis study aimed to assess the feasibility (adherence, acceptability, and data completeness) of a revised, structured LLM-based clinical reasoning laboratory (Laboratorio de Inteligencia Artificial [LIA]) in undergraduate physical therapy education and to explore its measurement properties, including its association with clinical judgment with a real patient assessed with the Lasater Clinical Judgment Rubric (LCJR).
MethodsIn this single-arm pilot feasibility study, 27 third-year physical therapy students at a Spanish university completed, in 2024-2025, 3 virtual patient cases (lumbar, cervical, and thoracic spine pain) simulated by ChatGPT (GPT-4) over 4 weeks, each twice. ChatGPT scored each attempt from 0 to 4 on the same 10 dimensions of the Lasater Clinical Judgment Rubric (LCJR; 0-40 points) that human assessors applied (1-4 per dimension; 10-40 points) during a real patient encounter. Feasibility outcomes were case completion, data completeness, and satisfaction (0-10). Measurement outcomes were internal consistency and convergent validity (Spearman correlations between model-generated and human LCJR scores). Variability estimates informed the sample size of a future trial.
ResultsAll 27 students were assessed with the LCJR; 22 (81%) completed at least 1 LIA case. Both attempts of the lumbar, cervical, and thoracic cases were completed by 78%, 63%, and 74% of students, respectively. Satisfaction data were available for 15 students (56%; mean 6.0, SD 1.3), and missing secondary data clustered in the third rotation. Internal consistency was excellent for the LCJR (=.96) and ranged from .43 to .90 for model-generated ratings. Model-generated LIA scores clustered near the maximum (mean 36.0, SD 1.4). Among completers, LIA scores correlated weakly with the LCJR ({rho}=0.25, 95% CI -0.20 to 0.61; {rho}=0.44, 95% CI 0.05-0.71, when noncompleted attempts were scored as 0). Grade point average correlated almost perfectly with the LCJR ({rho}=0.96; n=22); after adjustment for it, the LIA-LCJR association disappeared (partial {rho}=-0.10). A future trial would require approximately 39 students per group (80% power, conservative variance).
ConclusionsDelivering the LIA within the curriculum was feasible, but clinical judgment ratings generated by ChatGPT did not reflect those of human assessors with a real patient: they clustered near the maximum and showed no convergent validity once noncompletion was separated from performance. LLM-generated ratings should not be used as outcome or assessment measures without validation against human raters. Case calibration, completion tracking, and clarification of the GPA-LCJR relationship are also needed before a definitive trial.
Trial RegistrationClinicalTrials.gov NCT06809634; https://clinicaltrials.gov/study/NCT06809634
Related
- A Spanish-Language Version of the Emotion Reactivity Scale: Psychometric Properties, Measurement Invariance, and Criterion Validity in a Large Adolescent Sample
- Enhancing Evidence Translation Through Structured Clinical Reasoning: A Mixed-Methods Controlled Feasibility Study of Peer Visibility and Readiness Mechanisms in Applied Learning
- Influence of large language models on pediatricians’ clinical reasoning and decision-making process
- Adjudicator sensitivity of safety estimates in large language models for open-ended cardiovascular reasoning: a blinded multi-adjudicator benchmark study running title: adjudicator variability in LLM safety
- AI-Based Automated Segmentation of the Mandibular Cortex and Mental Foramen for Radiomorphometric Measurement on Panoramic Radiographs: A Feasibility Study
BibTeX
@misc{lerincalvo2026feasibility,
title = {{Feasibility and Measurement Properties of a Large Language Model-Based Virtual Patient Laboratory for Clinical Reasoning Training in Physical Therapy Students: Single-Arm Pilot Study}},
author = {A. Lerin-Calvo and A. Perez-Gonzalez and A. Sagarribay Villano and S. Di-Bonaventura and R. Ferrer-Pena},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.13.26364741},
url = {https://www.medrxiv.org/content/10.64898/2026.10.13.26364741v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Feasibility and Measurement Properties of a Large Language Model-Based Virtual Patient Laboratory for Clinical Reasoning Training in Physical Therapy Students: Single-Arm Pilot Study
AU - A. Lerin-Calvo
AU - A. Perez-Gonzalez
AU - A. Sagarribay Villano
AU - S. Di-Bonaventura
AU - R. Ferrer-Pena
PY - 2026
DA - 2026/10/13
PB - medRxiv
DO - 10.64898/2026.10.13.26364741
UR - https://www.medrxiv.org/content/10.64898/2026.10.13.26364741v1
AB - BackgroundLarge language models (LLMs) configured as virtual patients may expand structured practice of clinical reasoning in health professions education. Before testing effectiveness, it must be established whether students engage with them and whether the resulting scores are complete, reliable, and related to performance with real patients. Our earlier pilot randomized trial found very low engagement with a less structured LLM-based approach. ObjectiveThis study aimed to assess the feasibility (adherence, acceptability, and data completeness) of a revised, structured LLM-based clinical reasoning laboratory (Laboratorio de Inteligencia Artificial [LIA]) in undergraduate physical therapy education and to explore its measurement properties, including its association with clinical judgment with a real patient assessed with the Lasater Clinical Judgment Rubric (LCJR). MethodsIn this single-arm pilot feasibility study, 27 third-year physical therapy students at a Spanish university completed, in 2024-2025, 3 virtual patient cases (lumbar, cervical, and thoracic spine pain) simulated by ChatGPT (GPT-4) over 4 weeks, each twice. ChatGPT scored each attempt from 0 to 4 on the same 10 dimensions of the Lasater Clinical Judgment Rubric (LCJR; 0-40 points) that human assessors applied (1-4 per dimension; 10-40 points) during a real patient encounter. Feasibility outcomes were case completion, data completeness, and satisfaction (0-10). Measurement outcomes were internal consistency and convergent validity (Spearman correlations between model-generated and human LCJR scores). Variability estimates informed the sample size of a future trial. ResultsAll 27 students were assessed with the LCJR; 22 (81%) completed at least 1 LIA case. Both attempts of the lumbar, cervical, and thoracic cases were completed by 78%, 63%, and 74% of students, respectively. Satisfaction data were available for 15 students (56%; mean 6.0, SD 1.3), and missing secondary data clustered in the third rotation. Internal consistency was excellent for the LCJR (=.96) and ranged from .43 to .90 for model-generated ratings. Model-generated LIA scores clustered near the maximum (mean 36.0, SD 1.4). Among completers, LIA scores correlated weakly with the LCJR ({rho}=0.25, 95% CI -0.20 to 0.61; {rho}=0.44, 95% CI 0.05-0.71, when noncompleted attempts were scored as 0). Grade point average correlated almost perfectly with the LCJR ({rho}=0.96; n=22); after adjustment for it, the LIA-LCJR association disappeared (partial {rho}=-0.10). A future trial would require approximately 39 students per group (80% power, conservative variance). ConclusionsDelivering the LIA within the curriculum was feasible, but clinical judgment ratings generated by ChatGPT did not reflect those of human assessors with a real patient: they clustered near the maximum and showed no convergent validity once noncompletion was separated from performance. LLM-generated ratings should not be used as outcome or assessment measures without validation against human raters. Case calibration, completion tracking, and clarification of the GPA-LCJR relationship are also needed before a definitive trial. Trial RegistrationClinicalTrials.gov NCT06809634; https://clinicaltrials.gov/study/NCT06809634
ER -