{
  "abstract": "We investigated the potential utility of large language models (LLMs) in supporting patient safety efforts. Specifically, we evaluated the reasoning capabilities of LLMs in performing root cause analysis (RCA) of radiation oncology incidents using narrative reports from the Radiation Oncology Incident Learning System (RO-ILS). We prompted four state-of-the-art LLMs, Gemini 2.5 Pro, GPT-4o, o3, and Grok 3, with the “Background and Incident Overview” sections from 19 publicly available RO-ILS cases. Each model was instructed to perform RCA and generate root causes, lessons learned, and suggested actions using a standardized prompt based on AAPM RCA guidelines. Model outputs were evaluated using a combination of objective semantic similarity metrics (cosine similarity via Sentence Transformer), semi-subjective assessments (precision, recall, F1-score, expert-adjudicated PPV (Positive Predictive Value), hallucination rate and performance criteria including relevance, comprehensiveness, quality of justification and quality of solution), and subjective ratings (reasoning quality and overall performance) by five board-certified medical physicists. LLMs demonstrated satisfactory performance across evaluation metrics. All models exhibited some degree of hallucination, ranging from 11% to 61%. All the evaluated LLMs demonstrated comparable baseline capabilities in objective causal extraction, and Gemini 2.5 Pro exhibited the highest overall performance score among 4 models. Statistically significant differences were observed among models in expert-adjudicated PPV, hallucination rate, and subjective ratings (p < 0.05). LLMs delivered promising results as assistive tools for RCA in radiation oncology, with the ability to generate relevant and accurate analyses aligned with expert expectations. LLMs may support incident analysis and contribute to quality improvement efforts to advance patient safety in clinical radiation oncology practice. Author summary: Patient safety is paramount in radiation oncology treatment. This pilot study represents a pioneering investigation into the utility of Generative Artificial Intelligence (GAI) for enhancing critical safety mechanisms. Our research innovatively targets the sophisticated reasoning capabilities of Large Language Models (LLMs) in Root Cause Analysis (RCA), a complex, domain-specific process essential for preventing medical errors. We conducted a systematic evaluation of four advanced LLMs using real-world clinical incident reports to identify underlying systemic failures. Our results demonstrate that these AI models function as highly capable reasoning tools within clinical settings. All evaluated models received high validation from board-certified medical physicists. This innovation signals a paradigm shift in clinical implementation of digital health, proving that AI can successfully execute complex clinical reasoning tasks. By establishing a foundation for reliably augmenting human judgment in safety-critical domains, this proof-of-concept work provides a roadmap for integrating scalable, intelligent safety surveillance systems in radiation oncology, ultimately improving care for cancer patients.",
  "abstract_shown": true,
  "affiliations": [
    "Department of Radiation Oncology, University of Miami, Miami, Florida, United States of America"
  ],
  "article_type": "Research Article",
  "arxiv": null,
  "author_total": 6,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Yuntao Wang"
    },
    {
      "name": "Mariluz De Ornelas"
    },
    {
      "name": "Matthew T. Studenski"
    },
    {
      "name": "Elizabeth Bossart"
    },
    {
      "name": "Siamak P. Nejad-Davarani"
    },
    {
      "name": "Yunze Yang"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/36qhnc6srqwh4.bib",
  "cited_by": 0,
  "doi": "10.1371/journal.pdig.0001740",
  "fulltext_url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001740",
  "id": "36qhnc6srqwh4",
  "issns": [
    "2767-3170"
  ],
  "landing_url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001740",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "journal",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "openalex",
      "2718|Artificial Intelligence in Healthcare and Education"
    ],
    [
      "plos",
      "/Biology and life sciences/Neuroscience/Cognitive science/Cognitive psychology/Perception/Sensory perception/Hallucinations"
    ],
    [
      "plos",
      "/Biology and life sciences/Neuroscience/Cognitive science/Cognitive psychology/Reasoning"
    ],
    [
      "plos",
      "/Biology and life sciences/Neuroscience/Sensory perception/Hallucinations"
    ],
    [
      "plos",
      "/Biology and life sciences/Psychology/Cognitive psychology/Perception/Sensory perception/Hallucinations"
    ],
    [
      "plos",
      "/Biology and life sciences/Psychology/Cognitive psychology/Reasoning"
    ],
    [
      "plos",
      "/Medicine and health sciences/Clinical medicine/Clinical oncology/Radiation therapy"
    ],
    [
      "plos",
      "/Medicine and health sciences/Epidemiology/Medical risk factors/Cancer risk factors"
    ],
    [
      "plos",
      "/Medicine and health sciences/Oncology"
    ],
    [
      "plos",
      "/Medicine and health sciences/Oncology/Cancer risk factors"
    ],
    [
      "plos",
      "/Medicine and health sciences/Oncology/Cancer treatment"
    ],
    [
      "plos",
      "/Medicine and health sciences/Oncology/Cancer treatment/Radiation therapy"
    ],
    [
      "plos",
      "/Medicine and health sciences/Oncology/Clinical oncology/Radiation therapy"
    ],
    [
      "plos",
      "/People and places/Population groupings/Professions/Scientists/Physicists"
    ],
    [
      "plos",
      "/Physical sciences/Mathematics/Statistics/Similarity measures/Cosine similarity"
    ],
    [
      "plos",
      "/Science policy/Science and technology workforce/Careers in research/Scientists/Physicists"
    ],
    [
      "plos",
      "/Social sciences/Psychology/Cognitive psychology/Perception/Sensory perception/Hallucinations"
    ],
    [
      "plos",
      "/Social sciences/Psychology/Cognitive psychology/Reasoning"
    ]
  ],
  "openalex": "W7212256812",
  "pdf_url": "https://journals.plos.org/digitalhealth/article/file?id=10.1371/journal.pdig.0001740&type=printable",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-25",
  "publisher": "Public Library of Science",
  "ris": "https://research.draconitic.com/paper/36qhnc6srqwh4.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:17:43Z",
      "native_id": "10.1371/journal.pdig.0001740",
      "source": "plos",
      "url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001740"
    }
  ],
  "subjects": [
    {
      "field": "biology",
      "origin": "plos:neuroscience",
      "subfield": "neuroscience"
    },
    {
      "field": "medicine",
      "origin": "plos:oncology",
      "subfield": "oncology"
    }
  ],
  "title": "Augmenting patient safety surveillance in radiation oncology with large language model-based root cause analysis: A proof-of-concept study",
  "title_text": "Augmenting patient safety surveillance in radiation oncology with large language model-based root cause analysis: A proof-of-concept study",
  "type": "peer-reviewed",
  "updated_at": "2026-10-30T12:17:43Z",
  "url": "https://research.draconitic.com/paper/36qhnc6srqwh4",
  "venue": "PLOS Digital Health",
  "versions": []
}