PsyEval: a comprehensive large language model evaluation benchmark for mental health
5 affiliations
- 1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
- 2Department of Human Anatomy, Histology and Embryology, Peking University Health Science Center, Beijing, China
- 3Child Healthcare Department, Shanghai Children's Hospital, Shanghai, China
- 4ACL Lab, University of Texas at Arlington, Arlington, TX, USA. [email protected]
- 5X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China. [email protected]
Abstract
Evaluating large language models (LLMs) in the mental health domain presents distinct challenges due to the subtle, context-dependent, and subjective nature of psychological symptoms. We introduce PsyEval, a benchmark specifically designed to evaluate LLMs in mental health-related tasks across three core dimensions: knowledge, diagnosis, and emotional support. PsyEval is constructed to reflect the complexity of mental health scenarios and provides a structured framework for assessing model performance within this sensitive domain. Using PsyEval, we evaluate eleven advanced LLMs with different prompting strategies to investigate how prompting affects their responses. The results reveal considerable gaps in LLMs' current ability to reason accurately and respond appropriately in mental health contexts, while also indicating promising directions for future model enhancement.
Related
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- Comprehensive evaluation of large language models for pediatric pectus excavatum education
- Adjudicator sensitivity of safety estimates in large language models for open-ended cardiovascular reasoning: a blinded multi-adjudicator benchmark study running title: adjudicator variability in LLM safety
- Editorial: Innovating public mental health: integrating digital therapeutics for comprehensive mood disorder management
- The rise of small language models in healthcare: A comprehensive survey
BibTeX
@article{jin2026psyeval,
title = {{PsyEval: a comprehensive large language model evaluation benchmark for mental health}},
author = {Haoan Jin and Siyuan Chen and Dilawaier Dilixiati and Yewei Jiang and Kenny Q. Zhu and Mengyue Wu},
journal = {npj Mental Health Research},
year = {2026},
month = jul,
doi = {10.1038/s44184-026-00227-0},
url = {https://doi.org/10.1038/s44184-026-00227-0},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - PsyEval: a comprehensive large language model evaluation benchmark for mental health
AU - Haoan Jin
AU - Siyuan Chen
AU - Dilawaier Dilixiati
AU - Yewei Jiang
AU - Kenny Q. Zhu
AU - Mengyue Wu
PY - 2026
DA - 2026/07/09
JO - npj Mental Health Research
DO - 10.1038/s44184-026-00227-0
UR - https://doi.org/10.1038/s44184-026-00227-0
AB - Evaluating large language models (LLMs) in the mental health domain presents distinct challenges due to the subtle, context-dependent, and subjective nature of psychological symptoms. We introduce PsyEval, a benchmark specifically designed to evaluate LLMs in mental health-related tasks across three core dimensions: knowledge, diagnosis, and emotional support. PsyEval is constructed to reflect the complexity of mental health scenarios and provides a structured framework for assessing model performance within this sensitive domain. Using PsyEval, we evaluate eleven advanced LLMs with different prompting strategies to investigate how prompting affects their responses. The results reveal considerable gaps in LLMs' current ability to reason accurately and respond appropriately in mental health contexts, while also indicating promising directions for future model enhancement.
SN - 2731-4251
ER -