Decomposing deep neural network-brain representational similarity reveals distinct sources across the human visual hierarchy
Abstract
Deep neural networks predict neural responses across the visual hierarchy, yet published alignment scores do not reveal whether this correspondence reflects learned representations, architectural inductive biases, low-level image statistics, or categorical structure. We decompose DNN-brain alignment into four sources using variance partitioning on representational similarity analysis, applied to 7T fMRI from eight participants viewing approximately 10,000 natural images. The composition of alignment shifts systematically along the cortical hierarchy: in primary visual cortex, architecture and low-level statistics account for 46% of total explained variance, whereas in high-level visual cortex model-specific learned representations dominate, reaching 68% in the parahippocampal place area. The same framework applied to language models processing image captions reveals that alignment is strongly model-dependent: a sentence embedding model contributes 51% unique variance in higher visual cortex, whereas an autoregressive model contributes only 4%, with the remainder attributable to surface text statistics and category structure. Across nine models, alignment scores varied substantially in composition. Decomposing alignment into its sources is necessary before a score can be interpreted as evidence for convergent processing.
Related
- Stimulus symmetries can confound representational similarity analyses
- COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal
- Multidimensional semantic representations emerge from multi-frequency representational similarity learning
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks
- Assessing psychological, neural and AI-based representational geometries across multiple similarity properties
BibTeX
@article{li2026decomposing,
title = {{Decomposing deep neural network-brain representational similarity reveals distinct sources across the human visual hierarchy}},
author = {Xiaonai Li and Fangyao Zhang and Yuxuan Zhang},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-73438-2},
url = {https://www.nature.com/articles/s41598-026-73438-2},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Decomposing deep neural network-brain representational similarity reveals distinct sources across the human visual hierarchy
AU - Xiaonai Li
AU - Fangyao Zhang
AU - Yuxuan Zhang
PY - 2026
DA - 2026/10/01
JO - Scientific Reports
DO - 10.1038/s41598-026-73438-2
UR - https://www.nature.com/articles/s41598-026-73438-2
AB - Deep neural networks predict neural responses across the visual hierarchy, yet published alignment scores do not reveal whether this correspondence reflects learned representations, architectural inductive biases, low-level image statistics, or categorical structure. We decompose DNN-brain alignment into four sources using variance partitioning on representational similarity analysis, applied to 7T fMRI from eight participants viewing approximately 10,000 natural images. The composition of alignment shifts systematically along the cortical hierarchy: in primary visual cortex, architecture and low-level statistics account for 46% of total explained variance, whereas in high-level visual cortex model-specific learned representations dominate, reaching 68% in the parahippocampal place area. The same framework applied to language models processing image captions reveals that alignment is strongly model-dependent: a sentence embedding model contributes 51% unique variance in higher visual cortex, whereas an autoregressive model contributes only 4%, with the remainder attributable to surface text statistics and category structure. Across nine models, alignment scores varied substantially in composition. Decomposing alignment into its sources is necessary before a score can be interpreted as evidence for convergent processing.
SN - 2045-2322
ER -