Mental Health Literacy Across Psychology Students and Large Language Models
Abstract
Mental health misconceptions are widespread, yet it remains unclear how accurately psychology students distinguish such misconceptions from evidence-based information, whether this differs across stages of psychology education, and how their performance compares with that of large language models (LLMs). We evaluated the accuracy of 150 psychology students, including 70 first-semester and 80 more advanced students, in classifying 15 mental health myths and 15 facts. Additionally, we used Chat GPT-5.4, Claude Sonnet 4.5, DeepSeek V3.2, and GLM-5 to classify the same statements across 80 independent runs per statement and model. Students classified mental health myths more accurately than facts, and advanced students outperformed first-semester students in classifying both statement types (myths: 87.7% vs. 75.2%; facts: 69.3% vs. 55.4%). A binomial generalized linear mixed-effects model indicated that group differences in classification accuracy varied between myths and facts. LLMs generally showed higher accuracy than both student groups in classifying statements, although performance varied substantially across models, particularly for facts. Item-level analyses revealed partly distinct error patterns: students and LLMs struggled with some of the same facts, whereas they also showed relative advantages on specific statements. These findings suggest that mental health literacy differs significantly across stages of psychology education and that, although current LLMs perform strongly on this classification task, some item-specific errors recurred across repeated runs. Examining which misconceptions and facts remain difficult may help identify specific gaps in both psychology education and AI-generated mental health information.
Related
- Major differences in the relationship between electronic health literacy, social capital and negative mental health states among the university students in China
- Assessment of mental health literacy among non-medical undergraduate students of Makerere University, Uganda.
- Mental Health Literacy and its correlates among Zimbabwean Health Sciences students: a cross-sectional study and exploratory psychometric evaluation
- Health literacy, health status and health behavior among students and trainees in nursing: a longitudinal cohort study
- Family socioeconomic resources, family differentiation, and short-term change in health literacy among Chinese university students: a three-wave longitudinal study
BibTeX
@misc{richter2026mental,
title = {{Mental Health Literacy Across Psychology Students and Large Language Models}},
author = {Eileen Richter and Konrad Schöniger and Maya Schulze and Luisa Frede and Korbinian Moeller and Ronny Redlich and Markus Spitzer},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/e6s4t_v1},
url = {https://osf.io/preprints/psyarxiv/e6s4t_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Mental Health Literacy Across Psychology Students and Large Language Models
AU - Eileen Richter
AU - Konrad Schöniger
AU - Maya Schulze
AU - Luisa Frede
AU - Korbinian Moeller
AU - Ronny Redlich
AU - Markus Spitzer
PY - 2026
DA - 2026/09/21
PB - PsyArXiv
DO - 10.31234/osf.io/e6s4t_v1
UR - https://osf.io/preprints/psyarxiv/e6s4t_v1/
AB - Mental health misconceptions are widespread, yet it remains unclear how accurately psychology students distinguish such misconceptions from evidence-based information, whether this differs across stages of psychology education, and how their performance compares with that of large language models (LLMs). We evaluated the accuracy of 150 psychology students, including 70 first-semester and 80 more advanced students, in classifying 15 mental health myths and 15 facts. Additionally, we used Chat GPT-5.4, Claude Sonnet 4.5, DeepSeek V3.2, and GLM-5 to classify the same statements across 80 independent runs per statement and model. Students classified mental health myths more accurately than facts, and advanced students outperformed first-semester students in classifying both statement types (myths: 87.7% vs. 75.2%; facts: 69.3% vs. 55.4%). A binomial generalized linear mixed-effects model indicated that group differences in classification accuracy varied between myths and facts. LLMs generally showed higher accuracy than both student groups in classifying statements, although performance varied substantially across models, particularly for facts. Item-level analyses revealed partly distinct error patterns: students and LLMs struggled with some of the same facts, whereas they also showed relative advantages on specific statements. These findings suggest that mental health literacy differs significantly across stages of psychology education and that, although current LLMs perform strongly on this classification task, some item-specific errors recurred across repeated runs. Examining which misconceptions and facts remain difficult may help identify specific gaps in both psychology education and AI-generated mental health information.
ER -