Predictor Construction Can Reverse Multimodal Neural Contrasts
Abstract
Foundation-model features are increasingly used to ask what information neural activity represents, often by comparing prediction gains between nested encoding models. We show that such multimodal contrasts can change sign when only the conditioning predictor is reconstructed. Using fMRI from the Natural Scenes Dataset, DINOv2 visual features, and MPNet embeddings of MS COCO captions and Localized Narratives, a caption-narrative contrast in the additional predictive contribution of vision favors narratives when one short caption is compared with a long narrative (+0.012/+0.015 in Places), but favors captions after approximate word-count matching (-0.031/-0.023). The shift occurs across every measured ROI in both subjects and is driven primarily by differences in language-only prediction. Comparable contrasts also survive removal of image-specific content-word identity in several ROIs. These results show that nested neural contrasts do not identify represented content by themselves: predictor construction is part of the experimental design, and matched controls are required for representational claims.
Related
- Multimodal neural correlates of cognitive awareness in aging and Alzheimer’s disease
- Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language
- Host metabolic responses to SARS-CoV-2 and influenza viruses: parallels and contrasts
- From concentration to export: resource contrasts and bee traits shape pollinator spillover to crops
- From Contrasts to Representations: Rethinking fMRI Approaches for Cortico-Cerebellar Networks.
BibTeX
@misc{nadolskis2026predictor,
title = {{Predictor Construction Can Reverse Multimodal Neural Contrasts}},
author = {Lucas Nadolskis and Galen Pogoncheff and Michael Beyeler},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.16430},
eprint = {2609.16430},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.16430},
}
RIS
TY - UNPB
TI - Predictor Construction Can Reverse Multimodal Neural Contrasts
AU - Lucas Nadolskis
AU - Galen Pogoncheff
AU - Michael Beyeler
PY - 2026
DA - 2026/09/14
PB - arXiv
DO - 10.48550/arxiv.2609.16430
UR - https://arxiv.org/abs/2609.16430
AB - Foundation-model features are increasingly used to ask what information neural activity represents, often by comparing prediction gains between nested encoding models. We show that such multimodal contrasts can change sign when only the conditioning predictor is reconstructed. Using fMRI from the Natural Scenes Dataset, DINOv2 visual features, and MPNet embeddings of MS COCO captions and Localized Narratives, a caption-narrative contrast in the additional predictive contribution of vision favors narratives when one short caption is compared with a long narrative (+0.012/+0.015 in Places), but favors captions after approximate word-count matching (-0.031/-0.023). The shift occurs across every measured ROI in both subjects and is driven primarily by differences in language-only prediction. Comparable contrasts also survive removal of image-specific content-word identity in several ROIs. These results show that nested neural contrasts do not identify represented content by themselves: predictor construction is part of the experimental design, and matched controls are required for representational claims.
ER -