Language-model ratings of depression reflect the rater more than the patient
Abstract
Depression has no diagnostic blood test. Language models promise tireless, consistent assessment, but can accurate raters disagree about individuals? We pre-registered 880 language-model raters, crossing 11 open models with prompting and scoring choices, and applied them to 189 interviews against the eight-item Patient Health Questionnaire. Model choice explained 30.0% of summed-symptom score variance, stable participant differences 10.5%. Two randomly drawn raters with area under the receiver operating characteristic curve (AUC) >= 0.70 disagreed on screening decisions for 40% of participants, on average. Average over-rating governed how many were flagged, yet equal-capacity raters chose differently for about one participant in five. A locked analysis of 86 new interviews reproduced the main pre-registered findings. Exploratory recalibration with 40 labelled participants raised accuracy from about 60% to 75% and halved disagreement, leaving one participant in five decided differently. Calibration repaired much of the rater dependence without securing agreement about individuals.
Related
- Interpretable Symptom Vectors for Depression in a Large Language Model
- Early impressions count: Category ratings established from small samples.
- Assessing the Concordance Between LLM-Based ICD-10 Coding and a Drug Database for Antihypertensive Drug Contraindications: A Weighted Inter-Rater Agreement Study
- Human-like meaning maps from single-prompt VLM ratings of local scene meaning
- Subjective emotional ratings dissociate brain activation and network patterns during counterfactual outcome processing
BibTeX
@misc{lin2026languagemodel,
title = {{Language-model ratings of depression reflect the rater more than the patient}},
author = {Baihan Lin},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.08501},
eprint = {2610.08501},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.08501},
}
RIS
TY - UNPB
TI - Language-model ratings of depression reflect the rater more than the patient
AU - Baihan Lin
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.08501
UR - https://arxiv.org/abs/2610.08501
AB - Depression has no diagnostic blood test. Language models promise tireless, consistent assessment, but can accurate raters disagree about individuals? We pre-registered 880 language-model raters, crossing 11 open models with prompting and scoring choices, and applied them to 189 interviews against the eight-item Patient Health Questionnaire. Model choice explained 30.0% of summed-symptom score variance, stable participant differences 10.5%. Two randomly drawn raters with area under the receiver operating characteristic curve (AUC) >= 0.70 disagreed on screening decisions for 40% of participants, on average. Average over-rating governed how many were flagged, yet equal-capacity raters chose differently for about one participant in five. A locked analysis of 86 new interviews reproduced the main pre-registered findings. Exploratory recalibration with 40 labelled participants raised accuracy from about 60% to 75% and halved disagreement, leaving one participant in five decided differently. Calibration repaired much of the rater dependence without securing agreement about individuals.
ER -