Assessing User-Subjective Experience with Conversational Agents for Behavioral and Mental Health Interventions: Design and Psychometric Evaluation of CUE-Chat
Abstract
While AI-conversational agents are increasingly integrated into behavioral and mental health interventions, standardized instruments to assess their performance remain scarce. To address this gap, we developed an instrument to measure user-subjective experience with AI-driven agents in the healthcare domain. We focused on user perception due to its direct contribution to intervention effects, and on healthcare given that the rapid adoption of AI-agents in this domain continues to outpace the development of validated evaluation instruments. Across four studies, we developed and tested the Chatbot User Evaluation instrument (CUE-Chat). First, we identified relevant assessment domains and items through literature review, existing instruments, input from AI-driven agents, and the research team’s domain expertise. Following this, we requested feedback on draft subscales, definitions, and items from expert and laypeople panels. We then tested the structure of the instrument’s concept version. Finally, we conducted a psychometric evaluation and finalized the instrument. CUE-Chat consists of four factors: Perceived Efficacy (4 items), Usability (4 items), Intelligence (4 items), and Safety and Risk Management (4 items). Reliability and validity of the total score and each subscale was good to excellent (Cronbach’s α = .73-.92; Ω = .74-.93). Predictive validity was strong with all scales predictive of users’ overall rating of an AI-agent’s performance. We conclude that CUE-Chat’s strong psychometric properties are promising, and that it is particularly useful for researchers integrating AI-driven agents into behavioral and mental health interventions in both developmental and evaluative stages.
Related
- Subjective word-finding complaints in older adults: Development and preliminary psychometric evaluation of a novel self-reported questionnaire
- Psychometric evaluation of the Persian Version of the Postnatal Care Experience Scale (PCES) among Iranian postpartum women: a cross-sectional methodological study
- Subjective and Behavioral Difficulties in Social Contexts: Unique and Interacting Effects on Social Dysfunction Transdiagnostically
- Altered interoception in schizophrenia and its role in subjective emotional experience
- EEG signature in psychiatrist's subjective experience: a pilot study.
BibTeX
@misc{mertens2026assessing,
title = {{Assessing User-Subjective Experience with Conversational Agents for Behavioral and Mental Health Interventions: Design and Psychometric Evaluation of CUE-Chat}},
author = {Esther Mertens and Janis Butz and Jean-Louis van Gelder},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/8cbds_v2},
url = {https://osf.io/preprints/psyarxiv/8cbds_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Assessing User-Subjective Experience with Conversational Agents for Behavioral and Mental Health Interventions: Design and Psychometric Evaluation of CUE-Chat
AU - Esther Mertens
AU - Janis Butz
AU - Jean-Louis van Gelder
PY - 2026
DA - 2026/10/11
PB - PsyArXiv
DO - 10.31234/osf.io/8cbds_v2
UR - https://osf.io/preprints/psyarxiv/8cbds_v2/
AB - While AI-conversational agents are increasingly integrated into behavioral and mental health interventions, standardized instruments to assess their performance remain scarce. To address this gap, we developed an instrument to measure user-subjective experience with AI-driven agents in the healthcare domain. We focused on user perception due to its direct contribution to intervention effects, and on healthcare given that the rapid adoption of AI-agents in this domain continues to outpace the development of validated evaluation instruments. Across four studies, we developed and tested the Chatbot User Evaluation instrument (CUE-Chat). First, we identified relevant assessment domains and items through literature review, existing instruments, input from AI-driven agents, and the research team’s domain expertise. Following this, we requested feedback on draft subscales, definitions, and items from expert and laypeople panels. We then tested the structure of the instrument’s concept version. Finally, we conducted a psychometric evaluation and finalized the instrument. CUE-Chat consists of four factors: Perceived Efficacy (4 items), Usability (4 items), Intelligence (4 items), and Safety and Risk Management (4 items). Reliability and validity of the total score and each subscale was good to excellent (Cronbach’s α = .73-.92; Ω = .74-.93). Predictive validity was strong with all scales predictive of users’ overall rating of an AI-agent’s performance. We conclude that CUE-Chat’s strong psychometric properties are promising, and that it is particularly useful for researchers integrating AI-driven agents into behavioral and mental health interventions in both developmental and evaluative stages.
ER -