Evaluating ChatGPTs Responses to Chiropractic Patient Questions: A Comparative Study
- 1VA Puget Sound Health Care System
Abstract
Background Large language models (LLMs) are used at an increased frequency by patients to answer health related questions. Prior studies have evaluated LLMs and their performance with musculoskeletal (MSK) and spine domains, but have not been used to evaluate chiropractic-relevant content. Objective To evaluate ChatGPT responses to commonly asked chiropractic patient questions for concordance with chiropractic clinical practice guidelines. Methods Thirty questions addressing low back pain, neck pain, and chronic musculoskeletal pain were developed based on current U.S. chiropractic clinical practice guidelines (CPGs). ChatGPT (GPT-5.2) generated responses in isolated sessions. Responses were assessed for readability using established indices, then five licensed chiropractors independently rated each response using a 5-point Likert scale across four domains: relevance, clarity, completeness, and accuracy. Inter-rater reliability was evaluated using Krippendorffs alpha (ordinal). Results Readability measures demonstrated that ChatGPT responses were written at approximately a college-level reading grade. Expert ratings across all domain means (standard deviation) were predominately clustered at the upper end of rating scale (1-5): Relevance 4.72 {+/-} 0.56; Clarity 4.76 {+/-} 0.50; Completeness 4.23 {+/-} 0.81; Accuracy 4.00 {+/-} 1.02. The Krippendorffs alpha (ordinal) ranged from -0.023-0.214 across the domains, all reflective of poor inter-rater reliability. Conclusion ChatGPT responses were broadly consistent with chiropractic CPGs, though readability levels exceeded recommended patient education standards. While the expert panel frequently assigned similar scores, minor scoring differences and restricted score distributions produced low chance-corrected reliability. These findings suggest that ChatGPT can produce guideline concordant responses but readability and interpretive variability should be considered when integrating LLM generated information into patient education.
Related
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Evaluating large language models in patient education: a comparative analysis addressing frequently asked questions in peri-acetabular osteotomy
- Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
- Comparative evaluation of ChatGPT and gemini responses to patient-oriented questions on breast cancer
- Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.
BibTeX
@misc{pham2026evaluating,
title = {{Evaluating ChatGPTs Responses to Chiropractic Patient Questions: A Comparative Study}},
author = {A. V. Pham and A. C. Peterson and J. C. Tom and A. L. Jones and C. J. Daniels},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.25.26366458},
url = {https://www.medrxiv.org/content/10.64898/2026.10.25.26366458v1},
note = {CC0},
}
RIS
TY - UNPB
TI - Evaluating ChatGPTs Responses to Chiropractic Patient Questions: A Comparative Study
AU - A. V. Pham
AU - A. C. Peterson
AU - J. C. Tom
AU - A. L. Jones
AU - C. J. Daniels
PY - 2026
DA - 2026/10/25
PB - medRxiv
DO - 10.64898/2026.10.25.26366458
UR - https://www.medrxiv.org/content/10.64898/2026.10.25.26366458v1
AB - Background Large language models (LLMs) are used at an increased frequency by patients to answer health related questions. Prior studies have evaluated LLMs and their performance with musculoskeletal (MSK) and spine domains, but have not been used to evaluate chiropractic-relevant content. Objective To evaluate ChatGPT responses to commonly asked chiropractic patient questions for concordance with chiropractic clinical practice guidelines. Methods Thirty questions addressing low back pain, neck pain, and chronic musculoskeletal pain were developed based on current U.S. chiropractic clinical practice guidelines (CPGs). ChatGPT (GPT-5.2) generated responses in isolated sessions. Responses were assessed for readability using established indices, then five licensed chiropractors independently rated each response using a 5-point Likert scale across four domains: relevance, clarity, completeness, and accuracy. Inter-rater reliability was evaluated using Krippendorffs alpha (ordinal). Results Readability measures demonstrated that ChatGPT responses were written at approximately a college-level reading grade. Expert ratings across all domain means (standard deviation) were predominately clustered at the upper end of rating scale (1-5): Relevance 4.72 {+/-} 0.56; Clarity 4.76 {+/-} 0.50; Completeness 4.23 {+/-} 0.81; Accuracy 4.00 {+/-} 1.02. The Krippendorffs alpha (ordinal) ranged from -0.023-0.214 across the domains, all reflective of poor inter-rater reliability. Conclusion ChatGPT responses were broadly consistent with chiropractic CPGs, though readability levels exceeded recommended patient education standards. While the expert panel frequently assigned similar scores, minor scoring differences and restricted score distributions produced low chance-corrected reliability. These findings suggest that ChatGPT can produce guideline concordant responses but readability and interpretive variability should be considered when integrating LLM generated information into patient education.
ER -