Brain alignment of reasoning and action representations from vision-language and action models during naturalistic gameplay
Abstract
Understanding how humans and artificial intelligence systems predict and plan by interacting with their environment is a fundamental challenge at the intersection of neuroscience and machine learning. Most brain-encoding studies focus on aligning artificial models with brain activity during language comprehension or passive visual processing, while interactive brain alignment studies have to date been largely limited to reinforcement-learning (RL) agents and theory-based models. To address this gap, we study brain alignment of representative models from two foundation-model types, namely vision-language models (VLMs) and large-action models (LAMs), using fMRI recordings from participants playing naturalistic Atari-style video games. Specifically, we examine how action-focused and reasoning-focused prompts shape the models' internal representations and their alignment with fMRI brain activity. First, we find that both VLMs and LAMs achieve significantly higher voxel-wise encoding performance than RL baselines, with the advantage holding even under matched feature dimensionality. Second, compared to a no-prompt baseline, prompt-driven gains are larger in higher-order frontal-parietal and motor-planning regions than in early visual cortex, roughly 2-2.5× when averaged over region-of-interest (ROI) groups, although individual regions are heterogeneous. Third, variance partitioning reveals a qualitatively different representational organization. VLM representations are prompt-symmetric (12.4% unique action vs. 9.5% unique reasoning), whereas LAM representations are action-dominant (25.6% unique action vs.-8.2% unique reasoning), with the asymmetry strongest in frontal-motor cortex. Together, these results associate action specialization with distinct cortical alignment patterns in multimodal game-state representations, revealing differences hidden by similar prediction accuracy.
Related
- A Low-Rank Space Captures Linear Alignment Between LLM and Human Neural Activity During Naturalistic Language Listening
- Automated screening of autism via a pretrained vision-language model in naturalistic caregiver-child interactions.
- CLIVeg: Contrastive Image-Vegetation Alignment for Botany-Aware Earth Observation Representations
- Naturalistic language assessment and brain-network biomarkers in post-stroke aphasia for precision neurorehabilitation
- Experimental reasoning in large language models
BibTeX
@misc{oota2026brain,
title = {{Brain alignment of reasoning and action representations from vision-language and action models during naturalistic gameplay}},
author = {Subba Reddy Oota and Anant Khandelwal and Khushbu Pahwa and Satya Sai Srinath Namburi and Tanmoy Chakraborty and Bapi S. Raju and Manish Gupta},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.19352},
eprint = {2605.19352},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.19352},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Brain alignment of reasoning and action representations from vision-language and action models during naturalistic gameplay
AU - Subba Reddy Oota
AU - Anant Khandelwal
AU - Khushbu Pahwa
AU - Satya Sai Srinath Namburi
AU - Tanmoy Chakraborty
AU - Bapi S. Raju
AU - Manish Gupta
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2605.19352
UR - https://arxiv.org/abs/2605.19352
AB - Understanding how humans and artificial intelligence systems predict and plan by interacting with their environment is a fundamental challenge at the intersection of neuroscience and machine learning. Most brain-encoding studies focus on aligning artificial models with brain activity during language comprehension or passive visual processing, while interactive brain alignment studies have to date been largely limited to reinforcement-learning (RL) agents and theory-based models. To address this gap, we study brain alignment of representative models from two foundation-model types, namely vision-language models (VLMs) and large-action models (LAMs), using fMRI recordings from participants playing naturalistic Atari-style video games. Specifically, we examine how action-focused and reasoning-focused prompts shape the models' internal representations and their alignment with fMRI brain activity. First, we find that both VLMs and LAMs achieve significantly higher voxel-wise encoding performance than RL baselines, with the advantage holding even under matched feature dimensionality. Second, compared to a no-prompt baseline, prompt-driven gains are larger in higher-order frontal-parietal and motor-planning regions than in early visual cortex, roughly 2-2.5× when averaged over region-of-interest (ROI) groups, although individual regions are heterogeneous. Third, variance partitioning reveals a qualitatively different representational organization. VLM representations are prompt-symmetric (12.4% unique action vs. 9.5% unique reasoning), whereas LAM representations are action-dominant (25.6% unique action vs.-8.2% unique reasoning), with the asymmetry strongest in frontal-motor cortex. Together, these results associate action specialization with distinct cortical alignment patterns in multimodal game-state representations, revealing differences hidden by similar prediction accuracy.
ER -