Brain-IT-VQA: From Brain Signals to Answers
Abstract
Decoding visual content from fMRI signals recorded while a person views images, and specifically answering questions about the seen images, is a long-standing challenge. While significant progress has been made in recent years in visual question answering (VQA) from fMRI, performance remains limited. Moreover, although recent models can make increasingly accurate predictions, they have rarely been used as tools for understanding the structure of visual representations in the brain. We present Brain-IT-VQA, a framework for visual question answering from fMRI. Unlike previous methods, which extract a fixed representation from the fMRI signal, our extraction is conditioned on the question itself, so what is decoded from brain activity depends on what is being asked. Our model substantially outperforms previous fMRI-based captioning and VQA approaches. We further introduce NSD-VQA, a new dataset and benchmark for visual question answering from fMRI. Unlike existing image-fMRI VQA datasets, which typically provide only a few broad and weakly controlled questions per image, NSD-VQA provides on average 20 question-answer pairs per image across 20 controlled question categories that disentangle multiple levels of visual understanding. This enables more reliable and interpretable evaluation despite limited fMRI test data. Together, Brain-IT-VQA and NSD-VQA provide both a strong predictive framework and a tool for studying brain representations. Using this benchmark, we quantify which forms of visual and semantic information can be reliably decoded from fMRI responses to natural images. We further analyze the contributions of different brain regions across question types.
Related
- Neural coding of isopentyl acetate alarm pheromone signals in the honey bee brain
- Assessing whole-cortex excitability from electromagnetic brain signals
- The Fat-Body Secreted Neuropeptide CCHa2 Signals Insulin-Producing Cells in the Brain to Promote Sleep.
- When the Matrix Warns: Unraveling Brain Extracellular Matrix Components as Danger Signals in Glioma Progression
- Detecting movement preparation with HYDRE: a hybrid model of temporal and spectral brain signals
BibTeX
@misc{beliy2026brainitvqa,
title = {{Brain-IT-VQA: From Brain Signals to Answers}},
author = {Roman Beliy and Matias Cosarinsky and Oliver Heinimann and Navve Wasserman and Michal Irani},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.29588},
eprint = {2605.29588},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.29588},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - Brain-IT-VQA: From Brain Signals to Answers
AU - Roman Beliy
AU - Matias Cosarinsky
AU - Oliver Heinimann
AU - Navve Wasserman
AU - Michal Irani
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2605.29588
UR - https://arxiv.org/abs/2605.29588
AB - Decoding visual content from fMRI signals recorded while a person views images, and specifically answering questions about the seen images, is a long-standing challenge. While significant progress has been made in recent years in visual question answering (VQA) from fMRI, performance remains limited. Moreover, although recent models can make increasingly accurate predictions, they have rarely been used as tools for understanding the structure of visual representations in the brain. We present Brain-IT-VQA, a framework for visual question answering from fMRI. Unlike previous methods, which extract a fixed representation from the fMRI signal, our extraction is conditioned on the question itself, so what is decoded from brain activity depends on what is being asked. Our model substantially outperforms previous fMRI-based captioning and VQA approaches. We further introduce NSD-VQA, a new dataset and benchmark for visual question answering from fMRI. Unlike existing image-fMRI VQA datasets, which typically provide only a few broad and weakly controlled questions per image, NSD-VQA provides on average 20 question-answer pairs per image across 20 controlled question categories that disentangle multiple levels of visual understanding. This enables more reliable and interpretable evaluation despite limited fMRI test data. Together, Brain-IT-VQA and NSD-VQA provide both a strong predictive framework and a tool for studying brain representations. Using this benchmark, we quantify which forms of visual and semantic information can be reliably decoded from fMRI responses to natural images. We further analyze the contributions of different brain regions across question types.
ER -