Language-aligned models and structured scene descriptions reveal sensitivity to compositional scene structure in the high-level visual cortex
- 1Institute for Research in Fundamental Sciences (IPM)
Abstract
Natural scenes are defined not only by their contained objects, but also by the structured relations among those objects. Using 7T fMRI data from the Natural Scenes Dataset, we examined which information in scene-description representation contributes to the prediction of high-level visual cortical responses. We compared intact narratives with shuffled full description (bag-of-words), extracted content-word lists, and descriptions regenerated from those lists without access to the images of original narratives. Intact narrative descriptions outperformed the tested lexical controls. Both original and regenerated descriptions provided incremental predictive gains beyond bag-of-words, but the regenerated gain was only 21.3% as large as the original gain. Thus, constrained sentence reconstruction provided some additional predictive information without reproducing the original-description advantage. Exploratory image ratings indicated that this prediction advantage was greatest for scenes with richer scene compositional organization. Further, a language-aligned vision model provided additional predictive information beyond a self-supervised vision-only model, with a cortical distribution resembling the narrative advantage. These results address the representational basis of caption-based neural predictivity and are consistent with a contribution of structured scene information beyond simple entity co-occurrence. They further imply that language supervision may help induce comparable structure-sensitive representations in artificial vision models.
Related
- Host Identity and Urban Space Structure Phyllosphere Bacterial Communities Through Compositional Turnover in a Tropical Andean City
- Discovering compositional, interpretable structure in social behavior
- Anterior part of fronto-temporal language network as the neural basis of morphologically structured word form processing: evidence from Korean morphologically complex word.
- Large Language Models as Simulated Candidates in Objective Structured Clinical Examinations: A Rubric-Mapping Proof-of-Concept Study
- Structured biological evidence enables general-purpose language models to predict bacteriophage-host interactions
BibTeX
@misc{rajaei2026languagealigned,
title = {{Language-aligned models and structured scene descriptions reveal sensitivity to compositional scene structure in the high-level visual cortex}},
author = {K. Rajaei and A. Afshar and R. M. Cichy and H. Soltanian-Zadeh},
howpublished = {bioRxiv},
year = {2026},
month = jul,
doi = {10.64898/2026.07.27.740322},
url = {https://www.biorxiv.org/content/10.64898/2026.07.27.740322v2},
note = {CC BY},
}
RIS
TY - UNPB
TI - Language-aligned models and structured scene descriptions reveal sensitivity to compositional scene structure in the high-level visual cortex
AU - K. Rajaei
AU - A. Afshar
AU - R. M. Cichy
AU - H. Soltanian-Zadeh
PY - 2026
DA - 2026/07/27
PB - bioRxiv
DO - 10.64898/2026.07.27.740322
UR - https://www.biorxiv.org/content/10.64898/2026.07.27.740322v2
AB - Natural scenes are defined not only by their contained objects, but also by the structured relations among those objects. Using 7T fMRI data from the Natural Scenes Dataset, we examined which information in scene-description representation contributes to the prediction of high-level visual cortical responses. We compared intact narratives with shuffled full description (bag-of-words), extracted content-word lists, and descriptions regenerated from those lists without access to the images of original narratives. Intact narrative descriptions outperformed the tested lexical controls. Both original and regenerated descriptions provided incremental predictive gains beyond bag-of-words, but the regenerated gain was only 21.3% as large as the original gain. Thus, constrained sentence reconstruction provided some additional predictive information without reproducing the original-description advantage. Exploratory image ratings indicated that this prediction advantage was greatest for scenes with richer scene compositional organization. Further, a language-aligned vision model provided additional predictive information beyond a self-supervised vision-only model, with a cortical distribution resembling the narrative advantage. These results address the representational basis of caption-based neural predictivity and are consistent with a contribution of structured scene information beyond simple entity co-occurrence. They further imply that language supervision may help induce comparable structure-sensitive representations in artificial vision models.
ER -