Accuracy, Relevance, and Redundancy of ChatGPT-5.3 References in Open Fracture Management: A Methodological Analysis.
- 1Orthopedics and Traumatology, ÇOSB Kapaklı State Hospital, Tekirdağ, TUR.
- 2Orthopedics and Traumatology, Tekirdağ Namık Kemal University, Tekirdağ, TUR.
Abstract
Objective
This study aimed to evaluate the bibliographic accuracy of the references that ChatGPT-5.3 (OpenAI, San Francisco, CA) presented as PubMed-indexed sources for open fracture management and determine whether the verifiable references were relevant to the topics they were meant to support.Methodology
In a single session in March 2026, ChatGPT-5.3 (Plus subscription), with the web search function enabled, was asked to systematically describe the management of open fractures. The model was then prompted to provide three PubMed-indexed references for each topic it had generated. ChatGPT-5.3 structured open fracture management into 15 topic categories and produced a total of 45 references. Two independent reviewers assessed each reference for bibliographic accuracy and content relevance. Bibliographic accuracy was scored from 0 to 2, and content relevance from 0 to 1. Final scores were determined by consensus. Inter-rater agreement was assessed using Cohen's kappa coefficient.Results
Of the 45 references, 11 (24.4%) were fully bibliographically accurate. Nineteen references (42.2%) existed but contained errors in the PMID, year, journal, title, or other bibliographic details. Fifteen references (33.3%) could not be verified as valid PubMed-indexed articles. The overall bibliographic accuracy rate was 11 (45.6%), the content relevance rate was 30 (66.7%), and the overall performance score was 52.6%. All bibliographically verifiable references were content-relevant to their assigned open fracture management topics, with 30 (66.7%) references being content-relevant. The 45 references corresponded to only 36 unique sources; a single review article was reproduced under 8 (53.3%) of the 15 topic categories and accounted for 8 (72.7%) of the 11 fully accurate references. In a sensitivity analysis restricted to unique sources, the bibliographic accuracy rate decreased to 13 (37.5%). Inter-rater agreement was high, with Cohen's kappa values of 0.966 for bibliographic accuracy and 0.951 for content relevance.Conclusions
Although ChatGPT-5.3 generated a clinically reasonable and systematic framework for open fracture management, the references provided to support this response showed substantial bibliographic reliability problems. Because all verifiable references were content-relevant, the model's principal limitation appeared to lie not in the selection of irrelevant sources, but in the presentation of seemingly appropriate sources with inaccurate or unverifiable bibliographic information. References generated by ChatGPT should not be used for academic or clinical purposes without independent verification.
Related
- Transcriptomic Assessment of PD-L1 (CD274) in NSCLC: Biological Significance, Clinical Relevance, and Methodological Challenges
- Methodological considerations regarding the assessment of childhood fever management
- Focused ethnography protocol for exploring burn pain management practices in clinical settings: methodological considerations and implementation framework.
- Class-Discriminant Codebook Construction for Single-Token Signal Compression: A Pre-Registered Campaign Establishing a Statistically-Significant, Tax-Dependent Accuracy Lever, Its Rate-Relevance Frontier and Interior Optimum, and Its Threat- Model-Dependent Privacy Posture
- Age- and sex-specific normative references and performance zones for core muscular endurance in adolescents aged 12–16 years: a cross-sectional study
BibTeX
@article{kutluay2026accuracy,
title = {{Accuracy, Relevance, and Redundancy of ChatGPT-5.3 References in Open Fracture Management: A Methodological Analysis.}},
author = {Gündüz Ercan Kutluay and Yusuf Kondil},
journal = {Cureus},
year = {2026},
month = sep,
doi = {10.7759/cureus.116893},
url = {https://europepmc.org/article/MED/42928843},
note = {CC BY},
}
RIS
TY - JOUR
TI - Accuracy, Relevance, and Redundancy of ChatGPT-5.3 References in Open Fracture Management: A Methodological Analysis.
AU - Gündüz Ercan Kutluay
AU - Yusuf Kondil
PY - 2026
DA - 2026/09/25
JO - Cureus
DO - 10.7759/cureus.116893
UR - https://europepmc.org/article/MED/42928843
AB - Objective This study aimed to evaluate the bibliographic accuracy of the references that ChatGPT-5.3 (OpenAI, San Francisco, CA) presented as PubMed-indexed sources for open fracture management and determine whether the verifiable references were relevant to the topics they were meant to support.Methodology In a single session in March 2026, ChatGPT-5.3 (Plus subscription), with the web search function enabled, was asked to systematically describe the management of open fractures. The model was then prompted to provide three PubMed-indexed references for each topic it had generated. ChatGPT-5.3 structured open fracture management into 15 topic categories and produced a total of 45 references. Two independent reviewers assessed each reference for bibliographic accuracy and content relevance. Bibliographic accuracy was scored from 0 to 2, and content relevance from 0 to 1. Final scores were determined by consensus. Inter-rater agreement was assessed using Cohen's kappa coefficient.Results Of the 45 references, 11 (24.4%) were fully bibliographically accurate. Nineteen references (42.2%) existed but contained errors in the PMID, year, journal, title, or other bibliographic details. Fifteen references (33.3%) could not be verified as valid PubMed-indexed articles. The overall bibliographic accuracy rate was 11 (45.6%), the content relevance rate was 30 (66.7%), and the overall performance score was 52.6%. All bibliographically verifiable references were content-relevant to their assigned open fracture management topics, with 30 (66.7%) references being content-relevant. The 45 references corresponded to only 36 unique sources; a single review article was reproduced under 8 (53.3%) of the 15 topic categories and accounted for 8 (72.7%) of the 11 fully accurate references. In a sensitivity analysis restricted to unique sources, the bibliographic accuracy rate decreased to 13 (37.5%). Inter-rater agreement was high, with Cohen's kappa values of 0.966 for bibliographic accuracy and 0.951 for content relevance.Conclusions Although ChatGPT-5.3 generated a clinically reasonable and systematic framework for open fracture management, the references provided to support this response showed substantial bibliographic reliability problems. Because all verifiable references were content-relevant, the model's principal limitation appeared to lie not in the selection of irrelevant sources, but in the presentation of seemingly appropriate sources with inaccurate or unverifiable bibliographic information. References generated by ChatGPT should not be used for academic or clinical purposes without independent verification.
SN - 2168-8184
ER -