Toward a Validity Argument for Automated Assessment from Transcript and Video Data
Abstract
Automated assessment of psychological constructs from interview transcripts and video data has seen rapid technical development, yet the field lacks a systematic psychometric validity framework for interpreting scores from such measures. This paper applies Kane's interpretive/use argument structure alongside the five sources of validity evidence specified in the \textit{Standards for Educational and Psychological Testing} to automated transcript- and video-based assessment, using clinical assessment of depression as the primary illustrative case. We distinguish two design approaches --- item-based scoring, in which large language models (LLMs) rate explicit, theoretically derived items, and end-to-end embedding or classification approaches --- and examine how one could accrue validity evidence for each across the five sources. We argue that the field has concentrated almost exclusively on evidence based on relations to other variables (convergent validity against self-report criteria) while leaving response process and consequential evidence largely unaddressed. Further, we argue that construct validity evidence is often dismissed too quickly, especially for end-to-end approaches. Both evidence sources require fundamental reconceptualization for automated behavioral measures, and we propose reinterpretations suited to this context. Although the analysis centers on depression, the framework applies broadly to any construct assessed from passively collected video or transcript data. Implications for the design of future validation studies are discussed.
Related
- On the validity limits of the parametrisation method for invariant manifolds: an assessment of practical criteria for vibrating systems
- Confidence Filtering in Video-Based Gait Assessment: Measurement Error versus Selection Shift
- CRANDALL-PAZY RIGIDITY FOR LOWER SEMICONTINUOUS CONVEX FUNCTIONS: A BALANCED MEDIAN-FLOW ARGUMENT
- CRANDALL-PAZY RIGIDITY FOR C 1 CONVEX FUNCTIONS: A FIRST-ORDER MEDIAN-FLOW ARGUMENT
- A Diverging History Clock Is Not a Blow-up Proof: Lean-Checked Lemmas Showing Why a Section 10 Navier-Stokes Argument Does Not Certify Fefferman (C) or (D)
BibTeX
@misc{soland2026toward,
title = {{Toward a Validity Argument for Automated Assessment from Transcript and Video Data}},
author = {James Soland},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/e63gm_v1},
url = {https://osf.io/preprints/psyarxiv/e63gm_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Toward a Validity Argument for Automated Assessment from Transcript and Video Data
AU - James Soland
PY - 2026
DA - 2026/09/26
PB - PsyArXiv
DO - 10.31234/osf.io/e63gm_v1
UR - https://osf.io/preprints/psyarxiv/e63gm_v1/
AB - Automated assessment of psychological constructs from interview transcripts and video data has seen rapid technical development, yet the field lacks a systematic psychometric validity framework for interpreting scores from such measures. This paper applies Kane's interpretive/use argument structure alongside the five sources of validity evidence specified in the \textit{Standards for Educational and Psychological Testing} to automated transcript- and video-based assessment, using clinical assessment of depression as the primary illustrative case. We distinguish two design approaches --- item-based scoring, in which large language models (LLMs) rate explicit, theoretically derived items, and end-to-end embedding or classification approaches --- and examine how one could accrue validity evidence for each across the five sources. We argue that the field has concentrated almost exclusively on evidence based on relations to other variables (convergent validity against self-report criteria) while leaving response process and consequential evidence largely unaddressed. Further, we argue that construct validity evidence is often dismissed too quickly, especially for end-to-end approaches. Both evidence sources require fundamental reconceptualization for automated behavioral measures, and we propose reinterpretations suited to this context. Although the analysis centers on depression, the framework applies broadly to any construct assessed from passively collected video or transcript data. Implications for the design of future validation studies are discussed.
ER -