{
  "abstract": "Background: Peer review processes may inadequately assess compliance with established reporting guidelines such as the Consolidated Standards of Reporting Trials (CONSORT) criteria. Large language models (LLMs) demonstrate potential for systematic manuscript evaluation; however, how consistently different models assess adherence to CONSORT guidelines in published clinical trials remains unexplored. Methods: Twenty randomized controlled trials published in immunology journals between 2015 and 2016 were identified through PubMed. Three LLMs (ChatGPT-4o, Gemini 2.5 Flash, and Claude Sonnet 4.6) independently assessed compliance across 37 CONSORT 2010 subpoints. The primary endpoint was the difference between models in mean CONSORT compliance score. Secondary endpoints included inter-model agreement and the proportion of articles meeting a 90% compliance threshold. Statistical analysis employed repeated measures analysis of variance (ANOVA) with post-hoc pairwise comparisons (α = 0.05). Results: Mean CONSORT compliance rates were: ChatGPT-4o 80.8% (95% CI 75.8–85.8%), Gemini 2.5 Flash 64.4% (95% CI 58.0–70.8%), and Claude Sonnet 4.6 54.7% (95% CI 48.0–61.3%). Using a 90% compliance threshold as a quality benchmark, ChatGPT-4o identified 25% of papers (5/20) as meeting this standard, while Gemini 2.5 Flash and Claude Sonnet 4.6 each identified none (0/20) as meeting this standard. Repeated-measures ANOVA demonstrated significant differences between models (F(2,38) = 43.01, p < 0.001, partial η² = 0.694). All pairwise comparisons were statistically significant (ChatGPT-4o versus Gemini 2.5 Flash and ChatGPT-4o versus Claude Sonnet 4.6, both p < 0.001; Gemini 2.5 Flash versus Claude Sonnet 4.6, p = 0.014). Conclusions: LLMs varied substantially in their assessment of CONSORT compliance in published randomized trials, with a consistent ordering: ChatGPT-4o scored compliance highest, Gemini 2.5 Flash intermediate, and Claude Sonnet 4.6 lowest. Item-level agreement between models was only moderate, with no pairwise weighted kappa reaching the 0.61 substantial-agreement threshold. This inter-model variability indicates the need for standardized evaluation protocols before LLM-assisted manuscript screening is adopted.",
  "abstract_shown": true,
  "affiliations": [
    "Department of Medical Education and Clinical Sciences, Elson S. Floyd College of Medicine, Washington State University, Spokane, Washington, United States of America",
    "Department of Family Medicine, University of Washington School of Medicine, Seattle, Washington, United States of America"
  ],
  "article_type": "Research Article",
  "arxiv": null,
  "author_total": 3,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Daniel Y. Tsybulnik"
    },
    {
      "affs": [
        1
      ],
      "name": "Justin J. Gillette"
    },
    {
      "name": "Thomas F. Heston"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/1cnbj6ahp80ye.bib",
  "cited_by": 0,
  "doi": "10.1371/journal.pone.0358873",
  "fulltext_url": null,
  "id": "1cnbj6ahp80ye",
  "issns": [
    "1932-6203"
  ],
  "landing_url": "https://doi.org/10.1371/journal.pone.0358873",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "journal",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "openalex",
      "1804|Meta-analysis and systematic reviews"
    ],
    [
      "plos",
      "/Biology and life sciences/Immunology"
    ],
    [
      "plos",
      "/Biology and life sciences/Neuroscience/Cognitive science/Cognitive psychology/Language"
    ],
    [
      "plos",
      "/Biology and life sciences/Psychology/Cognitive psychology/Language"
    ],
    [
      "plos",
      "/Computer and information sciences/Computer applications/Web-based applications"
    ],
    [
      "plos",
      "/Medicine and health sciences/Clinical medicine/Clinical trials/Randomized controlled trials"
    ],
    [
      "plos",
      "/Medicine and health sciences/Immunology"
    ],
    [
      "plos",
      "/Medicine and health sciences/Medical humanities/Medical journals"
    ],
    [
      "plos",
      "/Medicine and health sciences/Pharmacology/Drug research and development/Clinical trials/Randomized controlled trials"
    ],
    [
      "plos",
      "/Physical sciences/Mathematics/Statistics/Statistical methods/Analysis of variance"
    ],
    [
      "plos",
      "/Research and analysis methods/Clinical trials/Randomized controlled trials"
    ],
    [
      "plos",
      "/Research and analysis methods/Mathematical and statistical techniques/Statistical methods/Analysis of variance"
    ],
    [
      "plos",
      "/Research and analysis methods/Research assessment/Peer review"
    ],
    [
      "plos",
      "/Research and analysis methods/Research assessment/Research reporting guidelines"
    ],
    [
      "plos",
      "/Social sciences/Psychology/Cognitive psychology/Language"
    ]
  ],
  "openalex": "W7214023326",
  "pdf_url": "https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0358873&type=printable",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-22",
  "publisher": "Public Library of Science",
  "ris": "https://research.draconitic.com/paper/1cnbj6ahp80ye.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:17:43Z",
      "native_id": "10.1371/journal.pone.0358873",
      "source": "plos",
      "url": "https://doi.org/10.1371/journal.pone.0358873"
    }
  ],
  "subjects": [
    {
      "field": "biology",
      "origin": "plos:immunology",
      "subfield": "immunology"
    },
    {
      "field": "mathematics",
      "origin": "plos:statistical methods",
      "subfield": "statistics"
    },
    {
      "field": "medicine",
      "origin": "plos:clinical trials",
      "subfield": "clinical-trials"
    }
  ],
  "title": "Variability among large language models in assessing CONSORT compliance of published randomized clinical trials",
  "title_text": "Variability among large language models in assessing CONSORT compliance of published randomized clinical trials",
  "type": "peer-reviewed",
  "updated_at": "2026-10-30T12:17:43Z",
  "url": "https://research.draconitic.com/paper/1cnbj6ahp80ye",
  "venue": "PLOS One",
  "versions": []
}