Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
Abstract
Objective Patients increasingly use large language models (LLMs) for menopause-related health information, yet the quality and readability of their responses have not been systematically compared across widely used platforms. This study compared the information quality and readability of responses from popular LLMs to frequently asked menopause questions. Methods In this comparative study, 50 menopause questions were compiled from four sources (ChatGPT, Google Gemini, Microsoft Copilot, and Google’s “People also ask”) after removing duplicates and less relevant items. Each was entered into the three tools, and responses were categorized with a modified Rothwell system and independently rated by two obstetrics and gynecology specialists using DISCERN, a Likert scale, PEMAT, Flesch Reading Ease, and Flesch-Kincaid Grade Level. Results Of the 50 questions, 70% were fact, 20% policy, and 10% value. Total DISCERN scores and overall information quality did not differ among platforms. Gemini scored higher on Likert and PEMAT actionability than ChatGPT and Copilot; ChatGPT and Gemini had higher Flesch Reading Ease than Copilot, which had a higher grade level. Overall readability differed (p = 0.022), with Gemini most often sufficient. Final overall sufficiency, meeting both information quality and readability, was low across all platforms (24.7%). Conclusion Popular LLMs produced menopause responses generally consistent with the literature but often limited in reliability structure and reading level, with low overall sufficiency across all three tools. Performance varied by platform and question type. Current consumer LLMs may complement rather than substitute for clinician-guided counseling; whether they can safely substitute requires patient-centered evaluation.
Related
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.
- Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk.
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
- Evaluating large language models in patient education: a comparative analysis addressing frequently asked questions in peri-acetabular osteotomy
BibTeX
@article{sancaktar2026quality,
title = {{Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study}},
author = {Çağlasu Sancaktar and Burak Çakmak},
journal = {Digital Health},
year = {2026},
month = oct,
doi = {10.1177/20552076261502183},
url = {https://doi.org/10.1177/20552076261502183},
note = {CC BY},
}
RIS
TY - JOUR
TI - Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
AU - Çağlasu Sancaktar
AU - Burak Çakmak
PY - 2026
DA - 2026/10/24
JO - Digital Health
DO - 10.1177/20552076261502183
UR - https://doi.org/10.1177/20552076261502183
AB - Objective Patients increasingly use large language models (LLMs) for menopause-related health information, yet the quality and readability of their responses have not been systematically compared across widely used platforms. This study compared the information quality and readability of responses from popular LLMs to frequently asked menopause questions. Methods In this comparative study, 50 menopause questions were compiled from four sources (ChatGPT, Google Gemini, Microsoft Copilot, and Google’s “People also ask”) after removing duplicates and less relevant items. Each was entered into the three tools, and responses were categorized with a modified Rothwell system and independently rated by two obstetrics and gynecology specialists using DISCERN, a Likert scale, PEMAT, Flesch Reading Ease, and Flesch-Kincaid Grade Level. Results Of the 50 questions, 70% were fact, 20% policy, and 10% value. Total DISCERN scores and overall information quality did not differ among platforms. Gemini scored higher on Likert and PEMAT actionability than ChatGPT and Copilot; ChatGPT and Gemini had higher Flesch Reading Ease than Copilot, which had a higher grade level. Overall readability differed (p = 0.022), with Gemini most often sufficient. Final overall sufficiency, meeting both information quality and readability, was low across all platforms (24.7%). Conclusion Popular LLMs produced menopause responses generally consistent with the literature but often limited in reliability structure and reading level, with low overall sufficiency across all three tools. Performance varied by platform and question type. Current consumer LLMs may complement rather than substitute for clinician-guided counseling; whether they can safely substitute requires patient-centered evaluation.
SN - 2055-2076
ER -