{
  "abstract": "Validation studies of zero-shot language-model scoring of psychological interviews often rely primarily on a single convergent correlation with a questionnaire to establish validity. However, a bivariate correlation cannot reveal whether a model lacks discriminant validity, agrees poorly in absolute terms, produces non-random missing scores, or scores inconsistently. We evaluated eight open-weight families (1–70B active parameters) and eight commercial frontier models. The models scored the same sets of clinical interviews (Study 1: suicide recovery, n = 149; Study 2: future self-continuity, n = 164) across 52 configurations. Convergent validity improves with model size in both studies. However, the standard open-weight models tested achieve only about three-quarters of frontier convergent validity, with 11-fold greater mean absolute bias, more format failures, and lower repeatability in Study 1. In Study 2, standard open models (≤ 27B) are as strongly correlated with a closely related construct (suicide recovery) as with the target (mean difference −0.014), although they correlate less strongly with a distal neighbor (life satisfaction). All six frontier generations correlate more strongly with the target than with either neighbor (mean difference from recovery +0.117). On a local workstation, Gemma 4 31B is within the frontier range on all discriminant validity indices, outperforming its similar-sized predecessor. Llama 3.3 70B shows no greater discriminant validity than Gemma 4 31B, suggesting that discriminant validity is a function of release, not parameter count alone. Based on these findings, we suggest a minimum validation standard across five domains.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": null,
  "author_total": 3,
  "authors": [
    {
      "name": "Yosef Sokol"
    },
    {
      "name": "Amar D. Mandavia"
    },
    {
      "name": "Jaclyn C. Kearns"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/7s418cbf85my7.bib",
  "cited_by": 0,
  "doi": "10.31234/osf.io/pwbhy_v1",
  "fulltext_url": null,
  "id": "7s418cbf85my7",
  "issns": [],
  "landing_url": "https://osf.io/preprints/psyarxiv/pwbhy_v1/",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "bepress",
      "Clinical Psychology"
    ],
    [
      "bepress",
      "Quantitative Methods"
    ],
    [
      "bepress",
      "Quantitative Psychology"
    ],
    [
      "bepress",
      "Social and Behavioral Sciences"
    ],
    [
      "openalex",
      "3207|Mental Health via Writing"
    ]
  ],
  "openalex": "W7223226170",
  "pdf_url": "https://osf.io/pwbhy/download",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-22",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/7s418cbf85my7.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:19:40Z",
      "native_id": "pwbhy",
      "source": "osf",
      "url": "https://osf.io/preprints/psyarxiv/pwbhy_v1/"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "bepress:Clinical Psychology",
      "subfield": "psychiatry"
    }
  ],
  "title": "Passing the Wrong Test: Model Size and Discriminant Validity in LLM-Based Psychological Assessment",
  "title_text": "Passing the Wrong Test: Model Size and Discriminant Validity in LLM-Based Psychological Assessment",
  "type": "preprint",
  "updated_at": "2026-10-30T12:19:40Z",
  "url": "https://research.draconitic.com/paper/7s418cbf85my7",
  "venue": "PsyArXiv",
  "versions": []
}