AI-Generated Treatment Recommendations in Newly Diagnosed Lung Cancer: A Single-Center Feasibility Study in a Resource-Limited Setting.
6 affiliations
- 1Department of Medical Oncology, Colentina Clinical Hospital, 020125 Bucharest, Romania.
- 2Faculty of Medicine, Titu Maiorescu University, 031593 Bucharest, Romania.
- 3Department of Medical Oncology, CF2 Clinical Hospital, 011464 Bucharest, Romania.
- 4Department of Pneumology, "Marius Nasta" Institute of Pneumology, 050159 Bucharest, Romania.
- 5Department of Pulmonology, "Carol Davila" University of Medicine and Pharmacy, 020021 Bucharest, Romania.
- 6Department of Dermatology, Colentina Clinical Hospital, 020125 Bucharest, Romania.
Abstract
Background
How treatment recommendations generated by large language models (LLMs) compare with real-world decisions in oncology remains poorly characterized, particularly in resource-limited settings.Methods
In this single-center feasibility study, four LLMs (ChatGPT, Claude, Gemini, and Grok, accessed via consumer interfaces in June 2026) generated treatment recommendations for 27 consecutive newly diagnosed lung cancer cases treated at a Romanian oncology unit (June 2024-May 2026). Each model's output and the treatment actually delivered were formatted identically (135 recommendations total), presented in randomized order, and scored for appropriateness (1-5) by a reviewer blinded to the source, with three quality flags. A second independent reviewer re-scored all recommendations to assess inter-rater reliability. Scores were compared using the Friedman test.Results
Mean appropriateness was highest for the real-world decision (4.52), followed by ChatGPT (4.48), Claude (4.44), Grok (4.41), and Gemini (4.37), with no significant difference (Friedman χ²₄ = 1.99, p = 0.74). No recommendation scored below 3. Inter-rater agreement was high in raw terms (93.3% within one point) but only fair after chance correction (weighted κ = 0.26), reflecting a ceiling effect.Conclusions
AI-generated recommendations were rated similarly to real-world decisions. The evaluation method proved feasible in a resource-limited setting; the small sample and ceiling effect preclude equivalence claims, and the findings are hypothesis-generating.
Related
- Delphi consensus guidelines for fungal sepsis in non-neutropenic critically ill patients in resource-limited settings: fungal infection study forum expert recommendations
- LLM-Generated Well-being Intervention Recommendations Are Sensitive to Psychological Profiles: A Proof-of-Concept Expert-Rating Study
- Protocol: WorkoutCPP - a pilot series of N-of-1 trials evaluating RL-generated adaptive exercise recommendations for pelvic pain management
- Silent Manipulation of Mental Health Treatment Recommendations From a Large Language Model
- Effectiveness of a Molecular Tumor Board in Making Personalized Treatment Recommendations for Patients with Advanced Cancer
BibTeX
@article{popescu2026aigenerated,
title = {{AI-Generated Treatment Recommendations in Newly Diagnosed Lung Cancer: A Single-Center Feasibility Study in a Resource-Limited Setting.}},
author = {Andrei Popescu and Mara Jidveian-Popescu and Beatrice Mahler and Rodica Olteanu and Emanuel Clem},
journal = {Current oncology (Toronto, Ont.)},
year = {2026},
month = oct,
doi = {10.3390/curroncol33100640},
url = {https://europepmc.org/article/MED/42930207},
note = {CC BY},
}
RIS
TY - JOUR
TI - AI-Generated Treatment Recommendations in Newly Diagnosed Lung Cancer: A Single-Center Feasibility Study in a Resource-Limited Setting.
AU - Andrei Popescu
AU - Mara Jidveian-Popescu
AU - Beatrice Mahler
AU - Rodica Olteanu
AU - Emanuel Clem
PY - 2026
DA - 2026/10/21
JO - Current oncology (Toronto, Ont.)
DO - 10.3390/curroncol33100640
UR - https://europepmc.org/article/MED/42930207
AB - Background How treatment recommendations generated by large language models (LLMs) compare with real-world decisions in oncology remains poorly characterized, particularly in resource-limited settings.Methods In this single-center feasibility study, four LLMs (ChatGPT, Claude, Gemini, and Grok, accessed via consumer interfaces in June 2026) generated treatment recommendations for 27 consecutive newly diagnosed lung cancer cases treated at a Romanian oncology unit (June 2024-May 2026). Each model's output and the treatment actually delivered were formatted identically (135 recommendations total), presented in randomized order, and scored for appropriateness (1-5) by a reviewer blinded to the source, with three quality flags. A second independent reviewer re-scored all recommendations to assess inter-rater reliability. Scores were compared using the Friedman test.Results Mean appropriateness was highest for the real-world decision (4.52), followed by ChatGPT (4.48), Claude (4.44), Grok (4.41), and Gemini (4.37), with no significant difference (Friedman χ²₄ = 1.99, p = 0.74). No recommendation scored below 3. Inter-rater agreement was high in raw terms (93.3% within one point) but only fair after chance correction (weighted κ = 0.26), reflecting a ceiling effect.Conclusions AI-generated recommendations were rated similarly to real-world decisions. The evaluation method proved feasible in a resource-limited setting; the small sample and ceiling effect preclude equivalence claims, and the findings are hypothesis-generating.
SN - 1198-0052
SN - 1718-7729
ER -