Language-Based Mental Health Assessments Do Not Perform Evenly Across Racial and Ethnic Groups
Abstract
Psycholinguistic tools that estimate psychological symptoms from language alone aim to provide unobtrusive assessments without needing questionnaires or clinicians. Researchers largely assume that these assessments function equivalently across demographic groups. We tested this assumption in a racially diverse sample of U.S. adults (N = 1,685; Black, Asian, Latino/Hispanic, White) who described their mental health and completed standard symptom questionnaires. Language-based markers of internalizing symptoms were often group-specific: First-person pronoun use predicted symptoms only among White participants, and sentiment varied in how strongly it tracked symptoms across groups. Machine learning models trained on White participants' language produced higher prediction error for other groups than those groups' own models did. Interpretability techniques revealed that White-trained models emphasized self-focused, agentic language more than minority groups' models. These findings indicate that linguistic tools cannot assume generalizability and must be evaluated for the populations they serve.
Related
- Same Name, Moving Target: Model Version Drift in Syntheses of Large Language Models for Mental Health.
- Maternal mental health impacts 2-year cognitive and language outcomes in infants born before 30 weeks of gestation
- Silent Manipulation of Mental Health Treatment Recommendations From a Large Language Model
- PsyEval: a comprehensive large language model evaluation benchmark for mental health
- Developmental Language Disorder and Mental Health Related GP Visits: Evidence from Health Records in England
BibTeX
@misc{mesquiti2026languagebased,
title = {{Language-Based Mental Health Assessments Do Not Perform Evenly Across Racial and Ethnic Groups}},
author = {Steven Mesquiti and Rebecca Carey and Nathaniel Daw and Erik Nook},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/fhbj7_v1},
url = {https://osf.io/preprints/psyarxiv/fhbj7_v1/},
note = {CC0},
}
RIS
TY - UNPB
TI - Language-Based Mental Health Assessments Do Not Perform Evenly Across Racial and Ethnic Groups
AU - Steven Mesquiti
AU - Rebecca Carey
AU - Nathaniel Daw
AU - Erik Nook
PY - 2026
DA - 2026/10/22
PB - PsyArXiv
DO - 10.31234/osf.io/fhbj7_v1
UR - https://osf.io/preprints/psyarxiv/fhbj7_v1/
AB - Psycholinguistic tools that estimate psychological symptoms from language alone aim to provide unobtrusive assessments without needing questionnaires or clinicians. Researchers largely assume that these assessments function equivalently across demographic groups. We tested this assumption in a racially diverse sample of U.S. adults (N = 1,685; Black, Asian, Latino/Hispanic, White) who described their mental health and completed standard symptom questionnaires. Language-based markers of internalizing symptoms were often group-specific: First-person pronoun use predicted symptoms only among White participants, and sentiment varied in how strongly it tracked symptoms across groups. Machine learning models trained on White participants' language produced higher prediction error for other groups than those groups' own models did. Interpretability techniques revealed that White-trained models emphasized self-focused, agentic language more than minority groups' models. These findings indicate that linguistic tools cannot assume generalizability and must be evaluated for the populations they serve.
ER -