COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal
Abstract
Language models can score psychiatric questionnaires from speech, but agreement with self-report may reflect the questionnaire rather than the person. We introduce psychometric representational similarity analysis, a framework for comparing the structure of speech-derived scores, self-report, item wording and theory, and implement it alongside person-level construct scoring in COMPASS 2.0. We show how similarly worded items induce covariance without psychological signal. In pre-registered discovery and confirmation analyses of clinical interviews from 275 participants, language-derived symptom geometry resembled wording more than self-report, with no structure beyond wording detected by the registered tests. Geometric agreement with self-report survived assigning participants someone else's answers, whereas person-paired scores captured distress more than specific symptoms. Complementary analyses examined counselling quality and wording structure across 34 instruments and the Research Domain Criteria (RDoC) framework. These findings distinguish agreement about psychological structure from evidence that language-derived assessments track individual people.
Related
- Stimulus symmetries can confound representational similarity analyses
- Multidimensional semantic representations emerge from multi-frequency representational similarity learning
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks
- Decomposing deep neural network-brain representational similarity reveals distinct sources across the human visual hierarchy
- Assessing psychological, neural and AI-based representational geometries across multiple similarity properties
BibTeX
@misc{lin2026compass,
title = {{COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal}},
author = {Baihan Lin},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06615},
eprint = {2610.06615},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.06615},
}
RIS
TY - UNPB
TI - COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal
AU - Baihan Lin
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.06615
UR - https://arxiv.org/abs/2610.06615
AB - Language models can score psychiatric questionnaires from speech, but agreement with self-report may reflect the questionnaire rather than the person. We introduce psychometric representational similarity analysis, a framework for comparing the structure of speech-derived scores, self-report, item wording and theory, and implement it alongside person-level construct scoring in COMPASS 2.0. We show how similarly worded items induce covariance without psychological signal. In pre-registered discovery and confirmation analyses of clinical interviews from 275 participants, language-derived symptom geometry resembled wording more than self-report, with no structure beyond wording detected by the registered tests. Geometric agreement with self-report survived assigning participants someone else's answers, whereas person-paired scores captured distress more than specific symptoms. Complementary analyses examined counselling quality and wording structure across 34 instruments and the Research Domain Criteria (RDoC) framework. These findings distinguish agreement about psychological structure from evidence that language-derived assessments track individual people.
ER -