Development and Preliminary Evaluation of the Patient-Artificial Intelligence Rating Instrument
- 1Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts; Center of Surgical and Medical Intelligence, Beth Israel Lahey Health, Bost
Abstract
ImportancePatients increasingly use artificial intelligence (AI) for health-related conversations, but evaluation measures largely reflect clinician judgment.
ObjectiveTo develop and preliminarily evaluate a patient instrument assessing the quality of patient-AI health conversations.
DesignCross-sectional survey conducted June 5-14, 2026, following patient-partnered instrument development and cognitive interviews.
SettingAn anonymous online survey.
ParticipantsA convenience sample of adults with prior use of large language models for personal health-related queries, recruited from an academic medical center-affiliated Insight Panel and patient peers.
ExposuresParticipants own prior health-related conversations with AI tools of their choosing.
Main Outcomes and MeasuresThe Patient-AI Rating instrument comprised 12 communication-quality and patient engagement items rated from 1 to 5 and a separately scored patient-perceived-harm item. Internal consistency was assessed using Cronbach , with sensitivity analyses using first ratings only for those using the tool to rate multiple conversations and participant-mean profiles. We examined item-total correlations, inter-item structure, and between- and within-participant variation.
ResultsFourteen participants contributed 46 complete ratings; among 13 providing demographics, median age was 63 years (IQR, 53-70 years), and 8 (62%) reported female gender. Participants provided a median of 2 ratings (range, 1-17). Cronbach was 0.88 (95% CI, 0.83-0.93) across all ratings, 0.85 using first ratings only, and 0.92 using participant-mean profiles. Corrected item-total correlations ranged from 0.42 to 0.72; mean inter-item correlation was 0.40. No item deletion improved Cronbach . The mean quality score was 4.43 (SD, 0.45). Among 9 participants contributing multiple ratings, the intraclass correlation coefficient was 0.45, suggesting inter-conversation variability. No perceived harm was reported in 44 of 46 ratings (96%); the harm item correlated minimally with the instrument total (r = 0.09).
Conclusions and RelevanceIn this small cross-sectional study of engaged patients, the Patient-AI Rating instrument showed promising internal consistency and variation across rated conversations. These findings support further evaluation of patient-reported communication quality alongside clinician-based measures. Validation in larger, more diverse populations, including assessment of construct and criterion validity, is needed before broader use.
Key PointsO_ST_ABSQuestionC_ST_ABSCan a patient instrument measure the quality of patients health-related conversations with artificial intelligence (AI)?
FindingsIn this cross-sectional study, 14 participants provided 46 ratings of their health-related AI conversations using a 12-item scale and a perceived-harm item (the Patient AI Rating (PAIR) instrument). The patient metric showed good internal consistency, and scores varied both by raters and across conversations rated by the same participant.
MeaningThe PAIR instrument offers a preliminary approach to incorporate patient perspective and experience into AI evaluation - a measure largely missing from current AI assessment; larger studies are needed to establish validity and generalizability.
Related
- Preliminary Clinical Evaluation in Cross‐Sectional Imaging: A Qualitative Inquiry Into the Ethical Considerations in the Era of Artificial Intelligence
- Development and preliminary psychometric evaluation of a modified Chinese instrument for self-rated mastery of diabetes exercise knowledge (DEKS-C)
- Development and Preliminary Evaluation of a Pharmacogenetic Panel for Risk Stratification of Liver Injury in Polypharmacy
- Development and Preliminary Psychometric Evaluation of a Questionnaire on Comprehensive Protection, Quality, and Humanization in Pediatric Hospital Care
- Subjective word-finding complaints in older adults: Development and preliminary psychometric evaluation of a novel self-reported questionnaire
BibTeX
@misc{rodman2026development,
title = {{Development and Preliminary Evaluation of the Patient-Artificial Intelligence Rating Instrument}},
author = {A. Rodman and E. Lowe and E. J. Thomas and Y. Benarroch and L. Ngo and T. Salant and F. C. Bourgeois and M. Newton and C. Walton and S. K. Bell},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.20.26366053},
url = {https://www.medrxiv.org/content/10.64898/2026.10.20.26366053v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Development and Preliminary Evaluation of the Patient-Artificial Intelligence Rating Instrument
AU - A. Rodman
AU - E. Lowe
AU - E. J. Thomas
AU - Y. Benarroch
AU - L. Ngo
AU - T. Salant
AU - F. C. Bourgeois
AU - M. Newton
AU - C. Walton
AU - S. K. Bell
PY - 2026
DA - 2026/10/20
PB - medRxiv
DO - 10.64898/2026.10.20.26366053
UR - https://www.medrxiv.org/content/10.64898/2026.10.20.26366053v1
AB - ImportancePatients increasingly use artificial intelligence (AI) for health-related conversations, but evaluation measures largely reflect clinician judgment. ObjectiveTo develop and preliminarily evaluate a patient instrument assessing the quality of patient-AI health conversations. DesignCross-sectional survey conducted June 5-14, 2026, following patient-partnered instrument development and cognitive interviews. SettingAn anonymous online survey. ParticipantsA convenience sample of adults with prior use of large language models for personal health-related queries, recruited from an academic medical center-affiliated Insight Panel and patient peers. ExposuresParticipants own prior health-related conversations with AI tools of their choosing. Main Outcomes and MeasuresThe Patient-AI Rating instrument comprised 12 communication-quality and patient engagement items rated from 1 to 5 and a separately scored patient-perceived-harm item. Internal consistency was assessed using Cronbach , with sensitivity analyses using first ratings only for those using the tool to rate multiple conversations and participant-mean profiles. We examined item-total correlations, inter-item structure, and between- and within-participant variation. ResultsFourteen participants contributed 46 complete ratings; among 13 providing demographics, median age was 63 years (IQR, 53-70 years), and 8 (62%) reported female gender. Participants provided a median of 2 ratings (range, 1-17). Cronbach was 0.88 (95% CI, 0.83-0.93) across all ratings, 0.85 using first ratings only, and 0.92 using participant-mean profiles. Corrected item-total correlations ranged from 0.42 to 0.72; mean inter-item correlation was 0.40. No item deletion improved Cronbach . The mean quality score was 4.43 (SD, 0.45). Among 9 participants contributing multiple ratings, the intraclass correlation coefficient was 0.45, suggesting inter-conversation variability. No perceived harm was reported in 44 of 46 ratings (96%); the harm item correlated minimally with the instrument total (r = 0.09). Conclusions and RelevanceIn this small cross-sectional study of engaged patients, the Patient-AI Rating instrument showed promising internal consistency and variation across rated conversations. These findings support further evaluation of patient-reported communication quality alongside clinician-based measures. Validation in larger, more diverse populations, including assessment of construct and criterion validity, is needed before broader use. Key PointsO_ST_ABSQuestionC_ST_ABSCan a patient instrument measure the quality of patients health-related conversations with artificial intelligence (AI)? FindingsIn this cross-sectional study, 14 participants provided 46 ratings of their health-related AI conversations using a 12-item scale and a perceived-harm item (the Patient AI Rating (PAIR) instrument). The patient metric showed good internal consistency, and scores varied both by raters and across conversations rated by the same participant. MeaningThe PAIR instrument offers a preliminary approach to incorporate patient perspective and experience into AI evaluation - a measure largely missing from current AI assessment; larger studies are needed to establish validity and generalizability.
ER -