Clinicians' Trust in AI-Based Clinical Recommendations Across Controlled Primary Care-Style Clinical Vignettes: Nurse-Dominant Experimental Pilot Study.
- 1Industrial and Management Systems Engineering, Benjamin M Statler College of Engineering and Mineral Resources, West Virginia University, Morgantown, WV, United States.
- 2Armstrong Institute Center for Health Care Human Factors, Johns Hopkins Medicine, Johns Hopkins University, Baltimore, MD, United States.
- 3Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States.
Abstract
Background
Safe integration of AI-enabled clinical decision support requires understanding whether users' trust and behavioral reliance are appropriately calibrated to recommendation quality.Objective
This pilot study examined a 3-item vignette-level trust construct and accept or reject behavior across sequential primary care-style clinical vignettes.Methods
A total of 68 health care professionals, including 59 (86.8%) registered nurses and 9 (13.2%) physicians, each evaluated 21 clinical vignettes from 1 of 2 series. Each series contained 10 correct and 11 intentionally incorrect AI recommendations. Participants accepted or rejected each recommendation and responded to 3 questionnaire items measuring trust, perceived transparency, and likelihood of acting on the recommendation before receiving correctness feedback and points. The unrounded item mean formed the trust composite. In the behavioral analysis, 50 unrecorded accept or reject responses were classified as rejections. A pooled cross-classified linear mixed-effects model was constructed to assess associations between the trust composite, recommendation correctness, vignette position, and participant and vignette characteristics. It included random intercepts for participant and vignette item, with case series included as a nuisance adjustment.Results
Participants accepted 374 of 748 (50%) incorrect recommendations and rejected 112 of 680 (16.47%) correct recommendations. Incorrect recommendations received lower prefeedback trust than correct recommendations (β=-0.772, 95% CI -1.036 to -0.507; standardized effect=-0.419). Trust declined modestly across vignette positions (β=-0.027, 95% CI -0.048 to -0.005; standardized effect=-0.088), with a decline in case series A but not case series B. Baseline intention to use AI was positively associated with trust (β=0.391, 95% CI 0.110-0.672; standardized effect=0.243), whereas higher perceived diagnostic difficulty was negatively associated with trust (β=-0.474, 95% CI -0.648 to -0.301; standardized effect=-0.257). In the exploratory lagged model, trust in the preceding recommendation was associated with trust in the current recommendation (β=0.266, 95% CI 0.210-0.322; standardized effect=0.264).Conclusions
Self-reported trust differentiated correct from incorrect recommendations in aggregate, while acceptance and rejection were not fully aligned with recommendation correctness. These preliminary findings identify a potential evaluation problem: lower trust in incorrect advice does not necessarily imply that users will reject it. The findings do not establish real-world clinical effects.
Related
- The impact of pain neuroscience education on physical therapy students' pain knowledge, attitudes, beliefs, and clinical recommendations: a randomized controlled trial.
- Effects of exercise interventions based on American College of Sports Medicine recommendations on physical function, body composition, and quality of life in older adults with sarcopenia: a meta-analysis of randomized controlled trials
- Auditing the Human LLM Autonomy Gap in Clinical Ethics: Development and Application of the Autonomy Index Across 50 Clinical Ethics Vignettes
- Self-Reflection Resilience-Recovery Activity Promotion Training for Emergency Medical Service Clinicians: A Randomized Controlled Pilot Trial of a Novel Digital Intervention
- Evaluating large language models as clinical decision support tools in primary healthcare settings: Protocol for a multi-country comparative validation study on expert-adjudicated hypothetical vignettes (hypMOOVE-PHC)
BibTeX
@article{choudhury2026clinicians,
title = {{Clinicians' Trust in AI-Based Clinical Recommendations Across Controlled Primary Care-Style Clinical Vignettes: Nurse-Dominant Experimental Pilot Study.}},
author = {Avishek Choudhury and Yeganeh Shahsavar and Ayse P Gurses},
journal = {JMIR human factors},
year = {2026},
month = oct,
doi = {10.2196/97649},
url = {https://europepmc.org/article/MED/42849041},
note = {CC BY},
}
RIS
TY - JOUR
TI - Clinicians' Trust in AI-Based Clinical Recommendations Across Controlled Primary Care-Style Clinical Vignettes: Nurse-Dominant Experimental Pilot Study.
AU - Avishek Choudhury
AU - Yeganeh Shahsavar
AU - Ayse P Gurses
PY - 2026
DA - 2026/10/08
JO - JMIR human factors
DO - 10.2196/97649
UR - https://europepmc.org/article/MED/42849041
AB - Background Safe integration of AI-enabled clinical decision support requires understanding whether users' trust and behavioral reliance are appropriately calibrated to recommendation quality.Objective This pilot study examined a 3-item vignette-level trust construct and accept or reject behavior across sequential primary care-style clinical vignettes.Methods A total of 68 health care professionals, including 59 (86.8%) registered nurses and 9 (13.2%) physicians, each evaluated 21 clinical vignettes from 1 of 2 series. Each series contained 10 correct and 11 intentionally incorrect AI recommendations. Participants accepted or rejected each recommendation and responded to 3 questionnaire items measuring trust, perceived transparency, and likelihood of acting on the recommendation before receiving correctness feedback and points. The unrounded item mean formed the trust composite. In the behavioral analysis, 50 unrecorded accept or reject responses were classified as rejections. A pooled cross-classified linear mixed-effects model was constructed to assess associations between the trust composite, recommendation correctness, vignette position, and participant and vignette characteristics. It included random intercepts for participant and vignette item, with case series included as a nuisance adjustment.Results Participants accepted 374 of 748 (50%) incorrect recommendations and rejected 112 of 680 (16.47%) correct recommendations. Incorrect recommendations received lower prefeedback trust than correct recommendations (β=-0.772, 95% CI -1.036 to -0.507; standardized effect=-0.419). Trust declined modestly across vignette positions (β=-0.027, 95% CI -0.048 to -0.005; standardized effect=-0.088), with a decline in case series A but not case series B. Baseline intention to use AI was positively associated with trust (β=0.391, 95% CI 0.110-0.672; standardized effect=0.243), whereas higher perceived diagnostic difficulty was negatively associated with trust (β=-0.474, 95% CI -0.648 to -0.301; standardized effect=-0.257). In the exploratory lagged model, trust in the preceding recommendation was associated with trust in the current recommendation (β=0.266, 95% CI 0.210-0.322; standardized effect=0.264).Conclusions Self-reported trust differentiated correct from incorrect recommendations in aggregate, while acceptance and rejection were not fully aligned with recommendation correctness. These preliminary findings identify a potential evaluation problem: lower trust in incorrect advice does not necessarily imply that users will reject it. The findings do not establish real-world clinical effects.
SN - 2292-9495
ER -