{
  "abstract": "ImportancePatients increasingly use artificial intelligence (AI) for health-related conversations, but evaluation measures largely reflect clinician judgment.\n\nObjectiveTo develop and preliminarily evaluate a patient instrument assessing the quality of patient-AI health conversations.\n\nDesignCross-sectional survey conducted June 5-14, 2026, following patient-partnered instrument development and cognitive interviews.\n\nSettingAn anonymous online survey.\n\nParticipantsA convenience sample of adults with prior use of large language models for personal health-related queries, recruited from an academic medical center-affiliated Insight Panel and patient peers.\n\nExposuresParticipants own prior health-related conversations with AI tools of their choosing.\n\nMain Outcomes and MeasuresThe Patient-AI Rating instrument comprised 12 communication-quality and patient engagement items rated from 1 to 5 and a separately scored patient-perceived-harm item. Internal consistency was assessed using Cronbach , with sensitivity analyses using first ratings only for those using the tool to rate multiple conversations and participant-mean profiles. We examined item-total correlations, inter-item structure, and between- and within-participant variation.\n\nResultsFourteen participants contributed 46 complete ratings; among 13 providing demographics, median age was 63 years (IQR, 53-70 years), and 8 (62%) reported female gender. Participants provided a median of 2 ratings (range, 1-17). Cronbach was 0.88 (95% CI, 0.83-0.93) across all ratings, 0.85 using first ratings only, and 0.92 using participant-mean profiles. Corrected item-total correlations ranged from 0.42 to 0.72; mean inter-item correlation was 0.40. No item deletion improved Cronbach . The mean quality score was 4.43 (SD, 0.45). Among 9 participants contributing multiple ratings, the intraclass correlation coefficient was 0.45, suggesting inter-conversation variability. No perceived harm was reported in 44 of 46 ratings (96%); the harm item correlated minimally with the instrument total (r = 0.09).\n\nConclusions and RelevanceIn this small cross-sectional study of engaged patients, the Patient-AI Rating instrument showed promising internal consistency and variation across rated conversations. These findings support further evaluation of patient-reported communication quality alongside clinician-based measures. Validation in larger, more diverse populations, including assessment of construct and criterion validity, is needed before broader use.\n\nKey PointsO_ST_ABSQuestionC_ST_ABSCan a patient instrument measure the quality of patients health-related conversations with artificial intelligence (AI)?\n\nFindingsIn this cross-sectional study, 14 participants provided 46 ratings of their health-related AI conversations using a 12-item scale and a perceived-harm item (the Patient AI Rating (PAIR) instrument). The patient metric showed good internal consistency, and scores varied both by raters and across conversations rated by the same participant.\n\nMeaningThe PAIR instrument offers a preliminary approach to incorporate patient perspective and experience into AI evaluation - a measure largely missing from current AI assessment; larger studies are needed to establish validity and generalizability.",
  "abstract_shown": true,
  "affiliations": [
    "Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts; Center of Surgical and Medical Intelligence, Beth Israel Lahey Health, Bost"
  ],
  "article_type": "PUBLISHAHEADOFPRINT",
  "arxiv": null,
  "author_total": 10,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "A. Rodman"
    },
    {
      "name": "E. Lowe"
    },
    {
      "name": "E. J. Thomas"
    },
    {
      "name": "Y. Benarroch"
    },
    {
      "name": "L. Ngo"
    },
    {
      "name": "T. Salant"
    },
    {
      "name": "F. C. Bourgeois"
    },
    {
      "name": "M. Newton"
    },
    {
      "name": "C. Walton"
    },
    {
      "name": "S. K. Bell"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/7qdk2d297f7t0.bib",
  "cited_by": 0,
  "doi": "10.64898/2026.10.20.26366053",
  "fulltext_url": "https://www.medrxiv.org/content/10.64898/2026.10.20.26366053v1.full",
  "id": "7qdk2d297f7t0",
  "issns": [],
  "landing_url": "https://www.medrxiv.org/content/10.64898/2026.10.20.26366053v1",
  "language": null,
  "licence": "CC-BY",
  "licence_label": "CC BY",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "medrxiv",
      "health informatics"
    ],
    [
      "openalex",
      "2718|Artificial Intelligence in Healthcare and Education"
    ]
  ],
  "openalex": "W7223222110",
  "pdf_url": "https://www.medrxiv.org/content/10.64898/2026.10.20.26366053v1.full.pdf",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-20",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/7qdk2d297f7t0.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:21:30Z",
      "native_id": "10.64898/2026.10.20.26366053",
      "source": "medrxiv",
      "url": "https://www.medrxiv.org/content/10.64898/2026.10.20.26366053v1"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "medrxiv:health informatics",
      "subfield": "epidemiology-public-health"
    }
  ],
  "title": "Development and Preliminary Evaluation of the Patient-Artificial Intelligence Rating Instrument",
  "title_text": "Development and Preliminary Evaluation of the Patient-Artificial Intelligence Rating Instrument",
  "type": "preprint",
  "updated_at": "2026-10-30T12:21:30Z",
  "url": "https://research.draconitic.com/paper/7qdk2d297f7t0",
  "venue": "medRxiv",
  "versions": []
}