Evaluation of the performance and temporal variability of large language models in patient education regarding pneumothorax: a seven-day analysis
- 1Department of Thoracic Surgery, Erciyes University Faculty of Medicine
- 2Department of Thoracic Surgery, Kayseri City Hospital
Abstract
This study investigates the readability, clinical reliability, and temporal consistency of artificial intelligence (AI) chatbots regarding pneumothorax information. A question bank comprising 40 patient-centered queries was deployed across three large language models (ChatGPT, Gemini, Copilot), stratified by two access tiers and two prompting strategies (zero-shot versus the optimized PROMPORT strategy). Queries were replicated longitudinally on Days 1, 3, and 7 under strict session-control protocols. Text accessibility was quantified using five automated readability indices, while two independent, blinded thoracic surgeons evaluated clinical quality using modified DISCERN (mDISCERN), JAMA benchmarks, and PEMAT-P indices. Readability metrics demonstrated absolute structural stability across the tracking intervals (p > 0.05). Unprompted configurations consistently generated complex, high-school-level outputs, whereas the PROMPORT strategy successfully compressed linguistic variances and neutralized chronological algorithmic drift (p > 0.05). Conversely, unprompted architectures exhibited significant temporal volatility in mDISCERN and JAMA profiles (p < 0.05), which was successfully stabilized by optimized prompt constraints. Inter-rater reliability was high across all structural evaluations. In conclusion, while unprompted models exhibit marked baseline linguistic and quality variations, the strategic integration of robust prompt engineering successfully enforces the temporal stability and clarity required for reliable digital public health communication.
Related
- A micro-learning-based education program about the principles of physical restraint using a mobile application and nursing students’ knowledge, attitudes, and performance regarding physical restraint for the elderly
- Peer Evaluation Rating Behavior in Differentiating Student Performance in Undergraduate Medical Education: An Observational Study
- Performance of five large language models in frozen shoulder patient education: a multidimensional evaluation of readability, quality, clinical alignment, and safety
- Comparison of Four Yogic Breathing Techniques Regarding Their Impact on Heart-Rate Variability and Breathing Frequency
- Fundamental data regarding the emergency management of infectious diseases in higher education institutions: HIV knowledge and stigma among college students in Southwest China
BibTeX
@article{onal2026evaluation,
title = {{Evaluation of the performance and temporal variability of large language models in patient education regarding pneumothorax: a seven-day analysis}},
author = {Ömer Önal and Suzan Temiz Bekce},
journal = {Scientific Reports},
year = {2026},
month = jul,
doi = {10.1038/s41598-026-62194-y},
url = {https://doi.org/10.1038/s41598-026-62194-y},
note = {CC BY},
}
RIS
TY - JOUR
TI - Evaluation of the performance and temporal variability of large language models in patient education regarding pneumothorax: a seven-day analysis
AU - Ömer Önal
AU - Suzan Temiz Bekce
PY - 2026
DA - 2026/07/01
JO - Scientific Reports
DO - 10.1038/s41598-026-62194-y
UR - https://doi.org/10.1038/s41598-026-62194-y
AB - This study investigates the readability, clinical reliability, and temporal consistency of artificial intelligence (AI) chatbots regarding pneumothorax information. A question bank comprising 40 patient-centered queries was deployed across three large language models (ChatGPT, Gemini, Copilot), stratified by two access tiers and two prompting strategies (zero-shot versus the optimized PROMPORT strategy). Queries were replicated longitudinally on Days 1, 3, and 7 under strict session-control protocols. Text accessibility was quantified using five automated readability indices, while two independent, blinded thoracic surgeons evaluated clinical quality using modified DISCERN (mDISCERN), JAMA benchmarks, and PEMAT-P indices. Readability metrics demonstrated absolute structural stability across the tracking intervals (p > 0.05). Unprompted configurations consistently generated complex, high-school-level outputs, whereas the PROMPORT strategy successfully compressed linguistic variances and neutralized chronological algorithmic drift (p > 0.05). Conversely, unprompted architectures exhibited significant temporal volatility in mDISCERN and JAMA profiles (p < 0.05), which was successfully stabilized by optimized prompt constraints. Inter-rater reliability was high across all structural evaluations. In conclusion, while unprompted models exhibit marked baseline linguistic and quality variations, the strategic integration of robust prompt engineering successfully enforces the temporal stability and clarity required for reliable digital public health communication.
SN - 2045-2322
ER -