Clarity Without Credibility? Human Versus AI Abstracts in Otolaryngology
8 affiliations
- 1City St. George's University London School of Medicine, Program Delivered by University of Nicosia at the Chaim Sheba Medical Center Ramat Gan Israel
- 2Department of Otolaryngology ‐ Head and Neck Surgery Stanford University School of Medicine Stanford California USA
- 3Department of Otolaryngology ‐ Head and Neck Surgery Medical University of Vienna Vienna Austria
- 4Department of Otorhinolaryngology Academic Hospital “Santa Maria della Misericordia”, Azienda Sanitaria Universitaria Friuli Centrale Udine Italy
- 5Hospices Civils de Lyon, Hôpital Lyon Sud, Service d'ORL, d'otoneurochirurgie et de Chirurgie Cervico‐faciale Pierre Bénite France
- 6Department of Otorhinolaryngology‐Head and Neck Surgery McGovern Medical School at the University of Texas Health Science Center Houston Texas USA
- 7Department of Ophthalmology University of Missouri Kansas City Kansas City Missouri USA
- 8Faculty of Medicine Tel Aviv University Tel Aviv Israel
Abstract
Objective This study evaluated whether otolaryngologists can distinguish between human‐ and machine‐written abstracts. The primary question was whether large language models (LLMs) produce abstracts comparable in clarity and usefulness to human‐authored work, and whether reviewers can identify authorship with accuracy. Methods A blinded cross‐sectional design was used. Forty‐eight abstracts were evaluated, consisting of twenty‐four human‐authored abstracts and 24 generated by four LLMs. Human abstracts were drawn from articles published after July 2025 to minimize overlap with LLM training data. Twenty otolaryngologists independently reviewed all abstracts. Using a structured rubric, raters classified authorship, rated clarity, usefulness, and confidence on 5‐point scales, and provided optional free‐text explanations. Group comparisons were performed using chi‐square and Mann–Whitney tests, with Kruskal–Wallis tests for model‐level analyses. Results Overall recognition accuracy was 44.7%. Human‐written abstracts were more often misclassified as AI than AI‐generated abstracts were mistaken for human. Human abstracts received significantly higher clarity and usefulness scores than LLM abstracts, though effect sizes were small. Confidence did not correlate with correctness, indicating miscalibration of rater judgments. Model‐level performance varied. Grok‐generated abstracts were most easily identified as AI, whereas GPT‐5 and Claude 3.5 more frequently resembled human writing. Free‐text rationales commonly referenced style, vagueness, or lack of detail when AI authorship was suspected. Conclusion LLMs generate abstracts that increasingly resemble human scientific writing, yet still lag in perceived usefulness and credibility. Clinicians were only moderately successful at detecting authorship and were frequently confident in incorrect classifications. These findings highlight both the promise and risks of AI‐assisted scientific communication.
Related
- AI versus human coding of NIH grant abstracts
- High- versus low-anthropomorphic-cue packages and health-information trust: social presence and source credibility under explicit AI disclosure
- Miniaturized Versus Standard-Tract Percutaneous Nephrolithotomy for Renal Calculi Larger Than 2 cm: Tract Caliber Determines Hemorrhagic Morbidity Without Compromising Stone Clearance
- Immune reconstitution and clinical outcomes in B-ALL patients with versus without pre-HSCT Immunotherapy
- Efficacy and safety of semaglutide versus placebo in adults with overweight or obesity without diabetes: updated meta-analysis of randomized trials
BibTeX
@article{hack2026clarity,
title = {{Clarity Without Credibility? Human Versus AI Abstracts in Otolaryngology}},
author = {Sholem Hack and Rebecca Attal and Lirit Levi and David T. Liu and Mahdokht Manavi and Luca Giovanni Locatello and Maxime Fieux and Evan M. deJoya and Joel M. Gutovitz and Shahar Eliahu and Tal Hefetz and Tomer Boldes and Ameen Biadsee},
journal = {World Journal of Otorhinolaryngology-Head and Neck Surgery},
year = {2026},
month = oct,
doi = {10.1002/wjo2.70104},
url = {https://doi.org/10.1002/wjo2.70104},
note = {CC BY},
}
RIS
TY - JOUR
TI - Clarity Without Credibility? Human Versus AI Abstracts in Otolaryngology
AU - Sholem Hack
AU - Rebecca Attal
AU - Lirit Levi
AU - David T. Liu
AU - Mahdokht Manavi
AU - Luca Giovanni Locatello
AU - Maxime Fieux
AU - Evan M. deJoya
AU - Joel M. Gutovitz
AU - Shahar Eliahu
AU - Tal Hefetz
AU - Tomer Boldes
AU - Ameen Biadsee
PY - 2026
DA - 2026/10/23
JO - World Journal of Otorhinolaryngology-Head and Neck Surgery
DO - 10.1002/wjo2.70104
UR - https://doi.org/10.1002/wjo2.70104
AB - Objective This study evaluated whether otolaryngologists can distinguish between human‐ and machine‐written abstracts. The primary question was whether large language models (LLMs) produce abstracts comparable in clarity and usefulness to human‐authored work, and whether reviewers can identify authorship with accuracy. Methods A blinded cross‐sectional design was used. Forty‐eight abstracts were evaluated, consisting of twenty‐four human‐authored abstracts and 24 generated by four LLMs. Human abstracts were drawn from articles published after July 2025 to minimize overlap with LLM training data. Twenty otolaryngologists independently reviewed all abstracts. Using a structured rubric, raters classified authorship, rated clarity, usefulness, and confidence on 5‐point scales, and provided optional free‐text explanations. Group comparisons were performed using chi‐square and Mann–Whitney tests, with Kruskal–Wallis tests for model‐level analyses. Results Overall recognition accuracy was 44.7%. Human‐written abstracts were more often misclassified as AI than AI‐generated abstracts were mistaken for human. Human abstracts received significantly higher clarity and usefulness scores than LLM abstracts, though effect sizes were small. Confidence did not correlate with correctness, indicating miscalibration of rater judgments. Model‐level performance varied. Grok‐generated abstracts were most easily identified as AI, whereas GPT‐5 and Claude 3.5 more frequently resembled human writing. Free‐text rationales commonly referenced style, vagueness, or lack of detail when AI authorship was suspected. Conclusion LLMs generate abstracts that increasingly resemble human scientific writing, yet still lag in perceived usefulness and credibility. Clinicians were only moderately successful at detecting authorship and were frequently confident in incorrect classifications. These findings highlight both the promise and risks of AI‐assisted scientific communication.
SN - 2589-1081
SN - 2095-8811
ER -