Beyond accuracy: a three-stage evaluation of large language models for postpartum health consultation and nursing decision support
- 1Delivery Center, Xi’an People’s Hospital (Xi’an Fourth Hospital)
- 2Women’s Health Care Center, Xi’an People’s Hospital (Xi’an Fourth Hospital)
Abstract
Background: Large language models may support postpartum health consultation and nursing care plan development, but their performance across different task types remains uncertain. This study compared the performance of ChatGPT 5.5 and DeepSeek V3 in these settings.
Methods: A three-stage cross-sectional evaluation was conducted, comprising repeated querying of 30 guideline-based questions, consultations involving 120 postpartum women, and 120 de-identified retrospective clinical cases. Responses were evaluated by clinical experts using a blinded assessment process. In Stage 2, the postpartum women who submitted the questions also performed blinded evaluations from the user perspective. Paired-samples t tests or Wilcoxon signed-rank tests were used, as appropriate. Holm-Bonferroni correction was applied within outcome families defined by study stage and evaluator type, and intraclass correlation coefficients were calculated to assess agreement among clinical expert ratings.
Results: In Stage 1, ChatGPT 5.5 received a higher correctness score than DeepSeek V3 (4.00 ± 0.00 versus 3.88 ± 0.33; Holm-adjusted p = 0.002). DeepSeek V3 received higher scores for completeness and clinical management appropriateness (both Holm-adjusted p < 0.001) and safety (Holm-adjusted p = 0.001), although its responses were more difficult to read. In Stage 2, ChatGPT 5.5 received higher patient ratings for satisfaction and usefulness and higher clinician ratings for correctness, completeness, helpfulness, and safety, whereas DeepSeek V3 received a higher clinician-rated readability score (all Holm-adjusted p < 0.001). In Stage 3, ChatGPT 5.5 received higher scores for clinical appropriateness, comprehensiveness, actionability, safety, and triage appropriateness (all Holm-adjusted p < 0.001), whereas DeepSeek V3 received a higher individualization score (Holm-adjusted p = 0.021).
Conclusion: LLM performance varied depending on the task, evaluation metric, and use case. ChatGPT 5.5 received higher scores across most clinically relevant dimensions in real-world postpartum consultations and nursing care plan generation, whereas DeepSeek V3 received higher scores for completeness and safety in standardized questions, clinician-rated readability in real-world consultations, and individualization in care-plan generation. LLMs may support health education and preparation of initial nursing care plan drafts, but their outputs require clinical review.
Related
- Multidimensional evaluation of generative AI chatbots for public consultation on sudden cardiac death: safety, accuracy, empathy, reliability, quality, and readability across six models
- Effect of In-line Clinical Decision Support to Improve Accuracy of American Society of Anesthesiologists Physical Status Documentation: A Two Center Evaluation of a Quality Intervention
- Citation Fabrication and Accuracy in AI-Generated Psychiatric Research: A Comparative Evaluation of Four Contemporary Large Language Models
- Nutrient Calculation Accuracy, Expert-Rated Clinical Appropriateness and Safety Indicators, and Reproducibility of Large Language Model-Generated Diet Plans for Polyendocrine Metabolic Ovarian Syndrome: A Real-Case-Based Evaluation
- Large language model-assisted CAD-RADS classification: structured report extraction, accuracy, and clinical decision-making risk of ChatGPT-4o/5 and DeepSeek-R1
BibTeX
@article{zhao2026beyond,
title = {{Beyond accuracy: a three-stage evaluation of large language models for postpartum health consultation and nursing decision support}},
author = {Xilin Zhao and Xinwen Zhang and Lei Liu},
journal = {Frontiers in Public Health},
year = {2026},
month = oct,
doi = {10.3389/fpubh.2026.1961647},
url = {https://www.frontiersin.org/articles/10.3389/fpubh.2026.1961647/full},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Beyond accuracy: a three-stage evaluation of large language models for postpartum health consultation and nursing decision support
AU - Xilin Zhao
AU - Xinwen Zhang
AU - Lei Liu
PY - 2026
DA - 2026/10/20
JO - Frontiers in Public Health
DO - 10.3389/fpubh.2026.1961647
UR - https://www.frontiersin.org/articles/10.3389/fpubh.2026.1961647/full
AB - Background: Large language models may support postpartum health consultation and nursing care plan development, but their performance across different task types remains uncertain. This study compared the performance of ChatGPT 5.5 and DeepSeek V3 in these settings. Methods: A three-stage cross-sectional evaluation was conducted, comprising repeated querying of 30 guideline-based questions, consultations involving 120 postpartum women, and 120 de-identified retrospective clinical cases. Responses were evaluated by clinical experts using a blinded assessment process. In Stage 2, the postpartum women who submitted the questions also performed blinded evaluations from the user perspective. Paired-samples t tests or Wilcoxon signed-rank tests were used, as appropriate. Holm-Bonferroni correction was applied within outcome families defined by study stage and evaluator type, and intraclass correlation coefficients were calculated to assess agreement among clinical expert ratings. Results: In Stage 1, ChatGPT 5.5 received a higher correctness score than DeepSeek V3 (4.00 ± 0.00 versus 3.88 ± 0.33; Holm-adjusted p = 0.002). DeepSeek V3 received higher scores for completeness and clinical management appropriateness (both Holm-adjusted p < 0.001) and safety (Holm-adjusted p = 0.001), although its responses were more difficult to read. In Stage 2, ChatGPT 5.5 received higher patient ratings for satisfaction and usefulness and higher clinician ratings for correctness, completeness, helpfulness, and safety, whereas DeepSeek V3 received a higher clinician-rated readability score (all Holm-adjusted p < 0.001). In Stage 3, ChatGPT 5.5 received higher scores for clinical appropriateness, comprehensiveness, actionability, safety, and triage appropriateness (all Holm-adjusted p < 0.001), whereas DeepSeek V3 received a higher individualization score (Holm-adjusted p = 0.021). Conclusion: LLM performance varied depending on the task, evaluation metric, and use case. ChatGPT 5.5 received higher scores across most clinically relevant dimensions in real-world postpartum consultations and nursing care plan generation, whereas DeepSeek V3 received higher scores for completeness and safety in standardized questions, clinician-rated readability in real-world consultations, and individualization in care-plan generation. LLMs may support health education and preparation of initial nursing care plan drafts, but their outputs require clinical review.
SN - 2296-2565
ER -