A Methodology to Evaluate the Efficacy and Patient-Centeredness of a Patient-Facing AI Chatbot: Development and Validation of a Standardized-Patient Method
- 1Vanderbilt University Medical Center
Abstract
BackgroundPatient-facing AI intake chatbots can enter clinical use without regulatory review, so their errors may first surface with actual patients.
ObjectiveTo develop and pilot a remote, voice-based standardized patient (SP) method for evaluating an intake chatbots history-taking, documentation, and patient-centeredness.
MethodsFour primary care cases had Easy and Difficult variants. In Phase 1, five SPs completed 40 encounters with internal medicine residents rating portrayal fidelity. In Phase 2, SPs completed 80 encounters with a ChatGPT 4o chatbot, recorded whether critical elements were asked about and addressed satisfactorily, and rated summaries; four debriefing interviews were analyzed. In Phase 3, two blinded clinicians rated transcripts and summaries from 101 encounters (32 resident-led, 69 chatbot-led).
ResultsPortrayal fidelity averaged 4.52 of 5. The chatbot asked about 59.3% of critical elements in Easy versus 50.2% in Difficult variants (adjusted OR 1.49; 95% CI, 1.15 to 1.93) and addressed 73.3% versus 64.6% (OR 1.47; 1.12 to 1.93); near-ceiling summary quality did not differ. SPs reported missed information and rushed pacing. Reviewer agreement was moderate to substantial for element judgments (AC1 0.49 to 0.70) but poor for totals (ICC 0.10 and 0.21). In exploratory comparisons, residents asked about more elements (adjusted difference 12.5 percentage points; 95% CI, 7.0 to 18.1) and satisfactorily addressed more (9.2 points; 3.8 to 14.6); totals did not differ.
ConclusionsSP portrayal was consistent enough for pre-clinical chatbot evaluation, and the method detected lower coverage in harder cases. Documentation ratings alone may miss conversational gaps, which require direct measurement.
Related
- Anatomofunctional Zone-Based Marking of the Lower Limbs (Angioderma Leg Mapping): A Proposed Standardized Methodology for Therapeutic Planning
- Next-generation digital polymerase chain reaction assay for BCR::ABL1 in chronic myeloid leukemia: comprehensive validation for standardized measurable residual disease detection
- Leveraging questionnaire-based physical activity levels and the Goldberg method to evaluate the biological plausibility of self-reported energy intake: a doubly labeled water validation study
- Accelerating drug development in infectious diseases using zebrafish disease models supported by pharmacokinetic pharmacodynamic modeling as new approach methodology (NAM)
- Quantitative Measurement of Hoover's Sign: Development of a Standardized Sensor, Interface Procedure, and Testing in Healthy Individuals
BibTeX
@misc{mamun2026methodology,
title = {{A Methodology to Evaluate the Efficacy and Patient-Centeredness of a Patient-Facing AI Chatbot: Development and Validation of a Standardized-Patient Method}},
author = {T. I. Mamun and A. R. Amogh and J. Rhodes and W. Martinez and S. Stern and K. Shah and K. B. Johnson and M. B. Weinger},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.08.26364702},
url = {https://www.medrxiv.org/content/10.64898/2026.10.08.26364702v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - A Methodology to Evaluate the Efficacy and Patient-Centeredness of a Patient-Facing AI Chatbot: Development and Validation of a Standardized-Patient Method
AU - T. I. Mamun
AU - A. R. Amogh
AU - J. Rhodes
AU - W. Martinez
AU - S. Stern
AU - K. Shah
AU - K. B. Johnson
AU - M. B. Weinger
PY - 2026
DA - 2026/10/08
PB - medRxiv
DO - 10.64898/2026.10.08.26364702
UR - https://www.medrxiv.org/content/10.64898/2026.10.08.26364702v1
AB - BackgroundPatient-facing AI intake chatbots can enter clinical use without regulatory review, so their errors may first surface with actual patients. ObjectiveTo develop and pilot a remote, voice-based standardized patient (SP) method for evaluating an intake chatbots history-taking, documentation, and patient-centeredness. MethodsFour primary care cases had Easy and Difficult variants. In Phase 1, five SPs completed 40 encounters with internal medicine residents rating portrayal fidelity. In Phase 2, SPs completed 80 encounters with a ChatGPT 4o chatbot, recorded whether critical elements were asked about and addressed satisfactorily, and rated summaries; four debriefing interviews were analyzed. In Phase 3, two blinded clinicians rated transcripts and summaries from 101 encounters (32 resident-led, 69 chatbot-led). ResultsPortrayal fidelity averaged 4.52 of 5. The chatbot asked about 59.3% of critical elements in Easy versus 50.2% in Difficult variants (adjusted OR 1.49; 95% CI, 1.15 to 1.93) and addressed 73.3% versus 64.6% (OR 1.47; 1.12 to 1.93); near-ceiling summary quality did not differ. SPs reported missed information and rushed pacing. Reviewer agreement was moderate to substantial for element judgments (AC1 0.49 to 0.70) but poor for totals (ICC 0.10 and 0.21). In exploratory comparisons, residents asked about more elements (adjusted difference 12.5 percentage points; 95% CI, 7.0 to 18.1) and satisfactorily addressed more (9.2 points; 3.8 to 14.6); totals did not differ. ConclusionsSP portrayal was consistent enough for pre-clinical chatbot evaluation, and the method detected lower coverage in harder cases. Documentation ratings alone may miss conversational gaps, which require direct measurement.
ER -