Accuracy and reproducibility of a large language model in assessing spirometry test acceptability and interpretation: a performance study.
5 affiliations
- 1Department of Respiratory Therapy, Faculty of Medical Rehabilitation Sciences, King Abdulaziz University, Jeddah, Saudi Arabia.
- 2Respiratory Therapy Unit, King Abdulaziz University Hospital, King Abdulaziz University, Jeddah, Saudi Arabia.
- 3Department of Pharmacology & Therapeutics, College of Medicine and Health Sciences, Arabian Gulf University, Manama, Bahrain.
- 4Arabian Gulf University, Manama, Bahrain.
- 5Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia.
Abstract
Spirometry is one of the most used pulmonary function tests for assessing respiratory function and diagnosing ventilatory disorders in clinical practice. Evaluation of test validity and interpretation is typically performed by trained healthcare professionals. The present study aims to evaluate the capability of a publicly available large language model (LLM) to perform quality assurance and interpretations of spirometry tests. A total of 150 spirometry tests were retrospectively collected from the pulmonary function laboratories. The spirometry tests were evaluated for acceptability and interpreted by clinicians according to ATS/ERS recommendations as part of standard clinical practice. Subsequently, a publicly available LLM (ChatGPT 5.5) was prompted to evaluate the acceptability and interpret the same spirometry tests according to ATS/ERS criteria. The model interpreted each spirometry test on three separate occasions for assessing reproducibility. Accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), with 95% confidence intervals (CIs), were calculated by comparing LLM assessments with clinician assessments of spirometry test acceptability and interpretation. For spirometry acceptability evaluation, ChatGPT demonstrated an overall accuracy of 87% (95% CI: 81%-92%), sensitivity of 91% (95% CI: 85%-95%), specificity of 61% (95% CI 39%-80%), PPV of 94% (95% CI: 89%-97%), and NPV of 48(95% CI: 29%-67%) with Cohen's k of 0.46. The spiromteric interpretation agreement with the clinicians reference standard was moderate, with exact agreement of 0.61 (95% CI: 0.54-0.69) while Cohen's Kappa (Unweighted) was 0.47 (95% CI: 0.37-0.56) and Macro F1 score was 0.39 (95% CI: 0.33-0.49). Reproducibility across three independent trials was moderate (Fleiss' κ 0.58). Class-specific interpretation performance varied by category; sensitivity was highest for normal pattern and restrictive defects (0.72), whereas sensitivity for obstructive defects was 0.27. F1-scores were highest for normal and restrictive defects (0.81 and 0.62, respectively). The current findings suggest that ChatGPT 5.5 has a limited capability, at its current stage of development, to serve as a reliable tool for independent spirometry interpretation in clinical practice. Nevertheless, it may serve as an adjunct to clinical decision-making and educational tools. Prospective controlled studies are needed to confirm these findings.
Related
- Reproducibility and diagnostic performance of ADC for assessing response to neoadjuvant immunotherapy in muscle-invasive bladder cancer: a multi-ROI comparison analysis from the phase-Ib NABUCCO trial
- Agreement between general practitioners and central reviewers on spirometry interpretation in primary care
- Diagnostic Accuracy of Tidal Breathing Patterns in Assessing Airflow Obstruction Compared with Spirometry.
- Reproducibility of Patch Test Reactions and Interpretation of Doubtful Results: A Retrospective Study of Identical Haptens Tested Simultaneously in Different Series
- Comparative validity and reproducibility of questionnaires in assessing types, frequency, and quantity of sweetened beverage consumption among Mahidol University students
BibTeX
@article{alotaibi2026accuracy,
title = {{Accuracy and reproducibility of a large language model in assessing spirometry test acceptability and interpretation: a performance study.}},
author = {Hajed M Al-Otaibi and Husam I Alahmadi and Malik A Althobiani and Kannan Sridharan and Abdullah A Alqarni and Ghazi A Alotaibi},
journal = {Frontiers in medicine},
year = {2026},
month = oct,
doi = {10.3389/fmed.2026.1974948},
url = {https://europepmc.org/article/MED/42952916},
note = {CC BY},
}
RIS
TY - JOUR
TI - Accuracy and reproducibility of a large language model in assessing spirometry test acceptability and interpretation: a performance study.
AU - Hajed M Al-Otaibi
AU - Husam I Alahmadi
AU - Malik A Althobiani
AU - Kannan Sridharan
AU - Abdullah A Alqarni
AU - Ghazi A Alotaibi
PY - 2026
DA - 2026/10/14
JO - Frontiers in medicine
DO - 10.3389/fmed.2026.1974948
UR - https://europepmc.org/article/MED/42952916
AB - Spirometry is one of the most used pulmonary function tests for assessing respiratory function and diagnosing ventilatory disorders in clinical practice. Evaluation of test validity and interpretation is typically performed by trained healthcare professionals. The present study aims to evaluate the capability of a publicly available large language model (LLM) to perform quality assurance and interpretations of spirometry tests. A total of 150 spirometry tests were retrospectively collected from the pulmonary function laboratories. The spirometry tests were evaluated for acceptability and interpreted by clinicians according to ATS/ERS recommendations as part of standard clinical practice. Subsequently, a publicly available LLM (ChatGPT 5.5) was prompted to evaluate the acceptability and interpret the same spirometry tests according to ATS/ERS criteria. The model interpreted each spirometry test on three separate occasions for assessing reproducibility. Accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), with 95% confidence intervals (CIs), were calculated by comparing LLM assessments with clinician assessments of spirometry test acceptability and interpretation. For spirometry acceptability evaluation, ChatGPT demonstrated an overall accuracy of 87% (95% CI: 81%-92%), sensitivity of 91% (95% CI: 85%-95%), specificity of 61% (95% CI 39%-80%), PPV of 94% (95% CI: 89%-97%), and NPV of 48(95% CI: 29%-67%) with Cohen's k of 0.46. The spiromteric interpretation agreement with the clinicians reference standard was moderate, with exact agreement of 0.61 (95% CI: 0.54-0.69) while Cohen's Kappa (Unweighted) was 0.47 (95% CI: 0.37-0.56) and Macro F1 score was 0.39 (95% CI: 0.33-0.49). Reproducibility across three independent trials was moderate (Fleiss' κ 0.58). Class-specific interpretation performance varied by category; sensitivity was highest for normal pattern and restrictive defects (0.72), whereas sensitivity for obstructive defects was 0.27. F1-scores were highest for normal and restrictive defects (0.81 and 0.62, respectively). The current findings suggest that ChatGPT 5.5 has a limited capability, at its current stage of development, to serve as a reliable tool for independent spirometry interpretation in clinical practice. Nevertheless, it may serve as an adjunct to clinical decision-making and educational tools. Prospective controlled studies are needed to confirm these findings.
SN - 2296-858X
ER -