Evidence of Limited Biases in Probed Language-based Assessments
Abstract
Large language models (LLMs) enable increasingly accurate language-based assessments (LBAs) of psychopathology. As these tools are more widely deployed in real-world settings, a critical question emerges: Do these tools work equally well for everyone? We examined whether probed LBAs, in which individuals respond to open-ended mental health questions in their own words, exhibit systematic biases that could undermine their clinical utility. We collected open-ended mental health descriptions from 1,260 participants alongside validated depression (PHQ-9) and anxiety (GAD-7) scores. We converted these descriptions to numeric representations using LLM-generated embeddings and applied ridge regression to predict symptom severity, achieving strong predictive accuracy (r > .60). We then examined whether assessment differences were systematically associated with sociodemographic factors (gender, age, education, employment, income) or linguistic characteristics. Our analysis revealed limited evidence of demographic or linguistic bias: most variables showed minimal association with assessment errors (i.e., the absolute differences between the LBAs and self reports). However, higher depression and anxiety severity were associated with larger absolute assessment differences (standardized β = .193, p = .038; β = .252, p = .003). We explored mitigation strategies, finding that downsampling training data eliminated the bias for depression and anxiety. Our findings indicate that probed language-based assessments exhibit limited bias overall, particularly regarding sociodemographic equity. However, the observed severity-related pattern highlights a fundamental challenge.
Related
- Language-Based Mental Health Assessments Do Not Perform Evenly Across Racial and Ethnic Groups
- Widespread Programs, Limited Evidence: How U.S. MD-Granting Medical Schools Define, Measure, and Evaluate Student Well-Being Efforts
- Accuracy Overstates Evidence Grounding and Abstention Reliability in Mammography Vision-Language Models
- CARE-MVLM: Counterfactual Abstention and Region-Grounded Evidence in Mammography Vision-Language Models
- Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
BibTeX
@misc{sparhuber2026evidence,
title = {{Evidence of Limited Biases in Probed Language-based Assessments}},
author = {Marc Sparhuber and Veerle Eijsbroek and Katarina Kjell and Salvatore Giorgi and Oscar Kjell},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/zrb3v_v1},
url = {https://osf.io/preprints/psyarxiv/zrb3v_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evidence of Limited Biases in Probed Language-based Assessments
AU - Marc Sparhuber
AU - Veerle Eijsbroek
AU - Katarina Kjell
AU - Salvatore Giorgi
AU - Oscar Kjell
PY - 2026
DA - 2026/09/03
PB - PsyArXiv
DO - 10.31234/osf.io/zrb3v_v1
UR - https://osf.io/preprints/psyarxiv/zrb3v_v1/
AB - Large language models (LLMs) enable increasingly accurate language-based assessments (LBAs) of psychopathology. As these tools are more widely deployed in real-world settings, a critical question emerges: Do these tools work equally well for everyone? We examined whether probed LBAs, in which individuals respond to open-ended mental health questions in their own words, exhibit systematic biases that could undermine their clinical utility. We collected open-ended mental health descriptions from 1,260 participants alongside validated depression (PHQ-9) and anxiety (GAD-7) scores. We converted these descriptions to numeric representations using LLM-generated embeddings and applied ridge regression to predict symptom severity, achieving strong predictive accuracy (r > .60). We then examined whether assessment differences were systematically associated with sociodemographic factors (gender, age, education, employment, income) or linguistic characteristics. Our analysis revealed limited evidence of demographic or linguistic bias: most variables showed minimal association with assessment errors (i.e., the absolute differences between the LBAs and self reports). However, higher depression and anxiety severity were associated with larger absolute assessment differences (standardized β = .193, p = .038; β = .252, p = .003). We explored mitigation strategies, finding that downsampling training data eliminated the bias for depression and anxiety. Our findings indicate that probed language-based assessments exhibit limited bias overall, particularly regarding sociodemographic equity. However, the observed severity-related pattern highlights a fundamental challenge.
ER -