Development and Preliminary Evaluation of a Brief Health Chatbot Acceptability Scale (CAS): A Pilot Study
- 1Nova Southeastern University
Abstract
Background. Conversational agents (chatbots) are increasingly used to deliver health information to middle-aged and older adults, a population in which the determinants of technology acceptance are known to diverge from those of younger users. However, acceptability is inconsistently defined and typically measured with ad hoc, unvalidated instruments. The psychometrically sound chatbot measures that exist assess usability rather than acceptability or were validated in student samples. Objective. To develop and conduct an initial psychometric evaluation of a brief, 10-item health chatbot acceptance measure grounded in the Technology Acceptance Model (TAM) and the Theory of Planned Behavior (TPB). Methods. Twenty adults (mean age = 52.8 years, range 28-81; 85% aged 40 or older; 75% female) enrolled in a sleep-application pilot usability study completed the 10 candidate items, the System Usability Scale (SUS), and sleep measures (PROMIS Sleep Disturbance 8a; Pittsburgh Sleep Quality Index). We examined item performance, internal consistency (Cronbach's alpha), dimensionality (exploratory factor analysis with oblique rotation), and convergent validity (correlations with the SUS). Results. A two-factor structure emerged and was invariant across rotation methods, resulting in an Information Quality factor and an Acceptance factor (interfactor r = .39). All items loaded [≥] .60 on their primary factor; mean communality was .83. Subscale reliabilities were high (Information Quality = .96; Acceptance = .94; full scale = .93). Both subscales correlated with the SUS in the expected direction (r = .69 and r = .59), providing preliminary support for convergent validity. Conclusions. The measure shows a coherent and theory-consistent two-factor structure and shows preliminary evidence of reliability and convergent validity. Given the small pilot sample, these results must be viewed as provisional but identify the measure as a candidate for formal validation in a larger, independent sample.
Related
- AI-based detection of suicidal ideation in text: model development and evaluation for a student mental health chatbot
- Adapting Non-Violent Resistance for Families of Young People with Complex Neurodevelopmental Needs: A Service Evaluation of Feasibility, Acceptability, and Preliminary Clinical Outcomes
- Feasibility, Acceptability, and Preliminary Outcomes of a Group-Based Brief Cognitive-Behavioral Therapy Protocol for Suicide Prevention in a Brazilian Public Mental Health Service: A Mixed-Methods Pilot Study.
- Development and Preliminary Evaluation of a Pharmacogenetic Panel for Risk Stratification of Liver Injury in Polypharmacy
- Development and Preliminary Psychometric Evaluation of a Questionnaire on Comprehensive Protection, Quality, and Humanization in Pediatric Hospital Care
BibTeX
@misc{ownby2026development,
title = {{Development and Preliminary Evaluation of a Brief Health Chatbot Acceptability Scale (CAS): A Pilot Study}},
author = {R. L. Ownby and J. Caballero and K. Thomas-Purcell},
howpublished = {medRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.05.26367380},
url = {https://www.medrxiv.org/content/10.64898/2026.11.05.26367380v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Development and Preliminary Evaluation of a Brief Health Chatbot Acceptability Scale (CAS): A Pilot Study
AU - R. L. Ownby
AU - J. Caballero
AU - K. Thomas-Purcell
PY - 2026
DA - 2026/11/05
PB - medRxiv
DO - 10.64898/2026.11.05.26367380
UR - https://www.medrxiv.org/content/10.64898/2026.11.05.26367380v1
AB - Background. Conversational agents (chatbots) are increasingly used to deliver health information to middle-aged and older adults, a population in which the determinants of technology acceptance are known to diverge from those of younger users. However, acceptability is inconsistently defined and typically measured with ad hoc, unvalidated instruments. The psychometrically sound chatbot measures that exist assess usability rather than acceptability or were validated in student samples. Objective. To develop and conduct an initial psychometric evaluation of a brief, 10-item health chatbot acceptance measure grounded in the Technology Acceptance Model (TAM) and the Theory of Planned Behavior (TPB). Methods. Twenty adults (mean age = 52.8 years, range 28-81; 85% aged 40 or older; 75% female) enrolled in a sleep-application pilot usability study completed the 10 candidate items, the System Usability Scale (SUS), and sleep measures (PROMIS Sleep Disturbance 8a; Pittsburgh Sleep Quality Index). We examined item performance, internal consistency (Cronbach's alpha), dimensionality (exploratory factor analysis with oblique rotation), and convergent validity (correlations with the SUS). Results. A two-factor structure emerged and was invariant across rotation methods, resulting in an Information Quality factor and an Acceptance factor (interfactor r = .39). All items loaded [≥] .60 on their primary factor; mean communality was .83. Subscale reliabilities were high (Information Quality = .96; Acceptance = .94; full scale = .93). Both subscales correlated with the SUS in the expected direction (r = .69 and r = .59), providing preliminary support for convergent validity. Conclusions. The measure shows a coherent and theory-consistent two-factor structure and shows preliminary evidence of reliability and convergent validity. Given the small pilot sample, these results must be viewed as provisional but identify the measure as a candidate for formal validation in a larger, independent sample.
ER -