Evaluating Open-Weight Large Language Models for Structured Depression Assessment from Clinical Interviews
Abstract
The administration of semi-structured clinical interviews for depression assessment is resource-intensive and susceptible to rater drift. While large language models (LLMs) offer potential for automated quality assurance, reliance on proprietary models creates data privacy barriers incompatible with secure clinical workflows. This study evaluates a privacy-preserving pipeline using open-weight LLMs to perform item-level scoring of the Montgomery-Åsberg Depression Rating Scale (MADRS) directly from interview transcripts. The sample included 541 video-recorded English-language interviews with 277 psychiatric inpatients diagnosed with diverse affective and psychotic disorders. Benchmarking 25 architectures, we found that an indirect scoring strategy, where models predict individual item scores that are subsequently summed, significantly outperformed direct total score prediction. This suggests that decomposing the assessment into intermediate reasoning steps improves performance. The best-performing models achieved “good” error rates and “excellent” correlations with trusted human labels, approaching established benchmarks for human inter-rater reliability. To determine the active ingredients of effective prompting, we conducted an ablation study. Results revealed that descriptive cues (scoring criteria) were the primary driver of performance, whereas few-shot examples provided negligible additional benefit, suggesting that zero-shot prompting is a viable, efficient strategy. However, error analysis identified a multimodal gap, where text-only models struggled more with items dependent on nonverbal behavior. We conclude that open-weight LLMs demonstrate strong potential to serve as secure “digital second opinions,” representing a promising step toward augmenting clinical decision-making without compromising patient privacy.
Related
- Large language models for depression assessment from brief daily diaries
- Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers
- Evaluating different large language model performances in evidence-based dental implantology: a blinded controlled experimental evaluation using expert panel assessment.
- Computational Linguistic Alignment in Psychosis from Semi-Structured Clinical Interviews
- Violence prevention and incident documentation in elderly and special needs care institutions in southern Germany: results from semi-structured interviews
BibTeX
@misc{girard2026evaluating,
title = {{Evaluating Open-Weight Large Language Models for Structured Depression Assessment from Clinical Interviews}},
author = {Jeffrey M. Girard and Gaossou Youssouf Kebe and Louis-Philippe Morency and Fernando De la Torre and Einat Liebenthal and Justin T. Baker},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/63sw4_v3},
url = {https://osf.io/preprints/psyarxiv/63sw4_v3/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evaluating Open-Weight Large Language Models for Structured Depression Assessment from Clinical Interviews
AU - Jeffrey M. Girard
AU - Gaossou Youssouf Kebe
AU - Louis-Philippe Morency
AU - Fernando De la Torre
AU - Einat Liebenthal
AU - Justin T. Baker
PY - 2026
DA - 2026/09/11
PB - PsyArXiv
DO - 10.31234/osf.io/63sw4_v3
UR - https://osf.io/preprints/psyarxiv/63sw4_v3/
AB - The administration of semi-structured clinical interviews for depression assessment is resource-intensive and susceptible to rater drift. While large language models (LLMs) offer potential for automated quality assurance, reliance on proprietary models creates data privacy barriers incompatible with secure clinical workflows. This study evaluates a privacy-preserving pipeline using open-weight LLMs to perform item-level scoring of the Montgomery-Åsberg Depression Rating Scale (MADRS) directly from interview transcripts. The sample included 541 video-recorded English-language interviews with 277 psychiatric inpatients diagnosed with diverse affective and psychotic disorders. Benchmarking 25 architectures, we found that an indirect scoring strategy, where models predict individual item scores that are subsequently summed, significantly outperformed direct total score prediction. This suggests that decomposing the assessment into intermediate reasoning steps improves performance. The best-performing models achieved “good” error rates and “excellent” correlations with trusted human labels, approaching established benchmarks for human inter-rater reliability. To determine the active ingredients of effective prompting, we conducted an ablation study. Results revealed that descriptive cues (scoring criteria) were the primary driver of performance, whereas few-shot examples provided negligible additional benefit, suggesting that zero-shot prompting is a viable, efficient strategy. However, error analysis identified a multimodal gap, where text-only models struggled more with items dependent on nonverbal behavior. We conclude that open-weight LLMs demonstrate strong potential to serve as secure “digital second opinions,” representing a promising step toward augmenting clinical decision-making without compromising patient privacy.
ER -