{
  "abstract": "Objective To compare three large language model (LLM) systems answering patient questions about renal cystic lesions in terms of medical content quality and radiologist-rated patient understandability. We also assessed inter-rater reliability and, exploratorily, the readability characteristics of English translations. Methods This cross-sectional, blinded, paired, multi-reader evaluation included 40 questions derived from authoritative patient education resources and web searches. Each question was submitted to OpenAI GPT-5.5, DeepSeek V4 Pro, and Ant A-Fu. Three radiologists independently rated every response on six 1–5 dimensions: medical accuracy, completeness, safety/risk guidance, radiologist-rated patient understandability, actionability/care advice, and overall quality. Overall comparisons used the Friedman test. Pairwise comparisons used paired Wilcoxon signed-rank tests with Holm correction. Intraclass correlation coefficients (ICCs) quantified inter-rater reliability. Sensitivity analyses evaluated all three responses and included 10,000 simulated random selections. Results No statistically significant overall differences were found in medical accuracy (4.65/4.68/4.65, P=0.790) or completeness (4.58/4.67/4.68, P=0.362). Safety/risk guidance also did not differ significantly (4.65/4.50/4.57, P=0.651). Differences were nonsignificant for actionability/care advice (4.34/4.42/4.38, P=0.120) and overall quality (4.30/4.38/4.40, P=0.134). Radiologist-rated patient understandability was the only dimension with an overall between-system difference (4.05/4.12/4.41, P<0.001; Kendall’s W=0.23). Ant A-Fu scored higher than OpenAI GPT-5.5 (Holm-adjusted P=0.0039; rank-biserial r=−0.768) and DeepSeek V4 Pro (Holm-adjusted P=0.0138; r=−0.638). This dimension showed very high inter-rater reliability [ICC(2,1)=0.947; ICC(2,3)=0.982]. The patient-understandability difference remained significant when all three responses were included and in all 10,000 random-selection simulations; the other five dimensions remained nonsignificant. Conclusion The primary analysis did not detect clear between-system differences in most medical-content dimensions. However, pronounced high-score clustering limited the discriminatory ability of the five-point rubric, and these nonsignificant findings do not establish similar or equivalent medical-content performance. Radiologist-rated patient understandability differed between systems and showed very high inter-rater reliability. Prospective studies involving patients are needed to determine whether this professional proxy predicts actual patient comprehension.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": null,
  "author_total": 8,
  "authors": [
    {
      "name": "Zhenghan Liu MM"
    },
    {
      "name": "Qianqian Chen MM"
    },
    {
      "name": "Tianxiao Hu BM"
    },
    {
      "name": "Mingyu Li MM"
    },
    {
      "name": "Haomei Yu MM"
    },
    {
      "name": "Jie Wang MD, PhD"
    },
    {
      "name": "Xiangming Fang MD, PhD"
    },
    {
      "name": "Xiaoyun Hu MD, PhD"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/2pq7dj7bdvp4b.bib",
  "cited_by": 0,
  "doi": "10.1177/20552076261504852",
  "fulltext_url": "https://doi.org/10.1177/20552076261504852",
  "id": "2pq7dj7bdvp4b",
  "issns": [
    "2055-2076"
  ],
  "landing_url": "https://doi.org/10.1177/20552076261504852",
  "language": "en",
  "licence": "CC-BY",
  "licence_label": "CC BY",
  "licence_source": "journal",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "lcc",
      "R858-859.7"
    ]
  ],
  "openalex": null,
  "pdf_url": null,
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-01",
  "publisher": "SAGE Publishing",
  "ris": "https://research.draconitic.com/paper/2pq7dj7bdvp4b.ris",
  "sources": [
    {
      "first_seen": "2026-11-02T14:21:15Z",
      "native_id": "0fa24d5bbbab4664ae18d62ae1e4adcb",
      "source": "doaj",
      "url": "https://doi.org/10.1177/20552076261504852"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "lcc:R858-859.7",
      "subfield": "general"
    }
  ],
  "title": "Medical content quality and radiologist-rated patient understandability of three large language model systems answering patient questions about renal cystic lesions: A blinded, Paired, multi-reader study",
  "title_text": "Medical content quality and radiologist-rated patient understandability of three large language model systems answering patient questions about renal cystic lesions: A blinded, Paired, multi-reader study",
  "type": "peer-reviewed",
  "updated_at": "2026-11-02T14:21:15Z",
  "url": "https://research.draconitic.com/paper/2pq7dj7bdvp4b",
  "venue": "Digital Health",
  "versions": []
}