Passing the Wrong Test: Model Size and Discriminant Validity in LLM-Based Psychological Assessment
Abstract
Validation studies of zero-shot language-model scoring of psychological interviews often rely primarily on a single convergent correlation with a questionnaire to establish validity. However, a bivariate correlation cannot reveal whether a model lacks discriminant validity, agrees poorly in absolute terms, produces non-random missing scores, or scores inconsistently. We evaluated eight open-weight families (1–70B active parameters) and eight commercial frontier models. The models scored the same sets of clinical interviews (Study 1: suicide recovery, n = 149; Study 2: future self-continuity, n = 164) across 52 configurations. Convergent validity improves with model size in both studies. However, the standard open-weight models tested achieve only about three-quarters of frontier convergent validity, with 11-fold greater mean absolute bias, more format failures, and lower repeatability in Study 1. In Study 2, standard open models (≤ 27B) are as strongly correlated with a closely related construct (suicide recovery) as with the target (mean difference −0.014), although they correlate less strongly with a distal neighbor (life satisfaction). All six frontier generations correlate more strongly with the target than with either neighbor (mean difference from recovery +0.117). On a local workstation, Gemma 4 31B is within the frontier range on all discriminant validity indices, outperforming its similar-sized predecessor. Llama 3.3 70B shows no greater discriminant validity than Gemma 4 31B, suggesting that discriminant validity is a function of release, not parameter count alone. Based on these findings, we suggest a minimum validation standard across five domains.
Related
- Sinicization of the Psychological Independence Scale and its reliability and validity testing among community⁃dwelling older adults in China
- Localized Debriefing Assessment for Simulation in Healthcare (DASH): Medical Student Engagement and Debriefing Assessment Validity
- Assessment in competency-based medical education for anesthesiology: validity evidence, equity, and faculty readiness
- The Master Adaptive Learner Behavioral Assessment Scale (MAL BAS): Development, Validity Evidence, Feasibility, and Acceptability of a New Instrument
- Heel-Rise Test for functional assessment in chronic venous insufficiency: a validity study Teste de elevação do calcanhar para avaliação funcional na insuficiência venosa crônica: um estudo de validação
BibTeX
@misc{sokol2026passing,
title = {{Passing the Wrong Test: Model Size and Discriminant Validity in LLM-Based Psychological Assessment}},
author = {Yosef Sokol and Amar D. Mandavia and Jaclyn C. Kearns},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/pwbhy_v1},
url = {https://osf.io/preprints/psyarxiv/pwbhy_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Passing the Wrong Test: Model Size and Discriminant Validity in LLM-Based Psychological Assessment
AU - Yosef Sokol
AU - Amar D. Mandavia
AU - Jaclyn C. Kearns
PY - 2026
DA - 2026/10/22
PB - PsyArXiv
DO - 10.31234/osf.io/pwbhy_v1
UR - https://osf.io/preprints/psyarxiv/pwbhy_v1/
AB - Validation studies of zero-shot language-model scoring of psychological interviews often rely primarily on a single convergent correlation with a questionnaire to establish validity. However, a bivariate correlation cannot reveal whether a model lacks discriminant validity, agrees poorly in absolute terms, produces non-random missing scores, or scores inconsistently. We evaluated eight open-weight families (1–70B active parameters) and eight commercial frontier models. The models scored the same sets of clinical interviews (Study 1: suicide recovery, n = 149; Study 2: future self-continuity, n = 164) across 52 configurations. Convergent validity improves with model size in both studies. However, the standard open-weight models tested achieve only about three-quarters of frontier convergent validity, with 11-fold greater mean absolute bias, more format failures, and lower repeatability in Study 1. In Study 2, standard open models (≤ 27B) are as strongly correlated with a closely related construct (suicide recovery) as with the target (mean difference −0.014), although they correlate less strongly with a distal neighbor (life satisfaction). All six frontier generations correlate more strongly with the target than with either neighbor (mean difference from recovery +0.117). On a local workstation, Gemma 4 31B is within the frontier range on all discriminant validity indices, outperforming its similar-sized predecessor. Llama 3.3 70B shows no greater discriminant validity than Gemma 4 31B, suggesting that discriminant validity is a function of release, not parameter count alone. Based on these findings, we suggest a minimum validation standard across five domains.
ER -