Evaluating cross-encoders for semantic similarity assessment in psychological questionnaires
Abstract
Correlations between rating scales are commonly interpreted as evidence of convergent or discriminant validity, yet prior studies suggest that part of these associations may be attributable to semantic similarity between item wordings rather than to genuine construct overlap alone. Building on this evidence, largely derived from bi-encoders, the present study explores whether cross-encoders, which jointly encode item pairs, offer a suitable technique for detecting semantic overlapping between questionnaire items. Using response data from the NEO-FFI and the PID5BF+M (N = 502, Labek et al., 2024), we examined whether cross-encoder-derived semantic similarity estimates are associated with empirical item correlations, and whether cross-encoders offer a systematic advantage over bi-encoders. Across twelve cross-encoder models, semantic distance was consistently negatively associated with absolute item correlations, reaching statistical significance in two-thirds of the models, with R2 values of up to .37. However, cross-encoders did not consistently outperform bi-encoders based on the same base models. These findings extend prior evidence for semantic components in scale intercorrelations to cross-encoder architectures, while indicating that predictive value depends more on model-specific training characteristics than on encoder architecture itself.
Related
- Constructing Composite Scores for Multidimensional Educational Assessment Batteries: Evaluating Alternative Psychometric Models in K–12 Reading Assessment
- Same path, different walk: a mechanistic comparison of looped and stacked transformer encoders on 12-lead ECG
- Task-Oriented Quantization for Quadratic Scheduling: Centroid Water-Filling and Power-Diagram Encoders
- It Takes Two Moments: Rigidity and Limits of Continuous Encoders for Over-the-Air Computation
- SiamJEPA: On the Role of Siamese Student Encoders in JEPA
BibTeX
@misc{kainz2026evaluating,
title = {{Evaluating cross-encoders for semantic similarity assessment in psychological questionnaires}},
author = {Isabella Kainz and Karin Labek and Roberto Viviani},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.22826},
eprint = {2608.22826},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.22826},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evaluating cross-encoders for semantic similarity assessment in psychological questionnaires
AU - Isabella Kainz
AU - Karin Labek
AU - Roberto Viviani
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2608.22826
UR - https://arxiv.org/abs/2608.22826
AB - Correlations between rating scales are commonly interpreted as evidence of convergent or discriminant validity, yet prior studies suggest that part of these associations may be attributable to semantic similarity between item wordings rather than to genuine construct overlap alone. Building on this evidence, largely derived from bi-encoders, the present study explores whether cross-encoders, which jointly encode item pairs, offer a suitable technique for detecting semantic overlapping between questionnaire items. Using response data from the NEO-FFI and the PID5BF+M (N = 502, Labek et al., 2024), we examined whether cross-encoder-derived semantic similarity estimates are associated with empirical item correlations, and whether cross-encoders offer a systematic advantage over bi-encoders. Across twelve cross-encoder models, semantic distance was consistently negatively associated with absolute item correlations, reaching statistical significance in two-thirds of the models, with R2 values of up to .37. However, cross-encoders did not consistently outperform bi-encoders based on the same base models. These findings extend prior evidence for semantic components in scale intercorrelations to cross-encoder architectures, while indicating that predictive value depends more on model-specific training characteristics than on encoder architecture itself.
ER -