Human similarity judgments reveal fine-grained representations of dynamic social scenes
Abstract
How does the human mind organize dynamic social scenes? Existing accounts rely on pre-defined social-affective or action-based taxonomies, yet whether humans use these dimensions to structure their representations of naturalistic social scenes remains unknown. We collected 53,223 human similarity judgments of videos depicting two-person actions and trained a sparse-positive-embedding model to recover the underlying latent dimensions. These learned dimensions explained twice as much variance in similarity judgments as pre-labeled annotations, captured interpretable fine-grained social-event structure (combining information about social relationships and actions), and fractionated broad semantic content (e.g., separating parent-child interactions into caregiving versus affectionate play). The same dimensions also predicted fMRI responses to passively viewed stimuli across visual cortex better than pre-labeled annotations, highlighting their importance beyond similarity judgments. Together, these results suggest that humans represent dynamic social scenes along fine-grained axes not captured by current taxonomies, and offer behaviorally-grounded targets for future studies of social vision.
Related
- Alignment of visual brain encoding models with individual perceptual similarity judgments
- Perceptual similarity representations beyond the sensory cortex
- Multidimensional semantic representations emerge from multi-frequency representational similarity learning
- On the role of geometric representations for ultra-fine-grained crop variety identification
- Protein novelty, sequence similarity, and negative sampling controls reveal benchmark specific discrimination in protein protein interactionprediction
BibTeX
@misc{garcia2026human,
title = {{Human similarity judgments reveal fine-grained representations of dynamic social scenes}},
author = {Kathy Garcia and Leyla Isik},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/8cn9f_v2},
url = {https://osf.io/preprints/psyarxiv/8cn9f_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Human similarity judgments reveal fine-grained representations of dynamic social scenes
AU - Kathy Garcia
AU - Leyla Isik
PY - 2026
DA - 2026/09/30
PB - PsyArXiv
DO - 10.31234/osf.io/8cn9f_v2
UR - https://osf.io/preprints/psyarxiv/8cn9f_v2/
AB - How does the human mind organize dynamic social scenes? Existing accounts rely on pre-defined social-affective or action-based taxonomies, yet whether humans use these dimensions to structure their representations of naturalistic social scenes remains unknown. We collected 53,223 human similarity judgments of videos depicting two-person actions and trained a sparse-positive-embedding model to recover the underlying latent dimensions. These learned dimensions explained twice as much variance in similarity judgments as pre-labeled annotations, captured interpretable fine-grained social-event structure (combining information about social relationships and actions), and fractionated broad semantic content (e.g., separating parent-child interactions into caregiving versus affectionate play). The same dimensions also predicted fMRI responses to passively viewed stimuli across visual cortex better than pre-labeled annotations, highlighting their importance beyond similarity judgments. Together, these results suggest that humans represent dynamic social scenes along fine-grained axes not captured by current taxonomies, and offer behaviorally-grounded targets for future studies of social vision.
ER -