CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps
- 1University of California, San Diego
Abstract
Recent advances in brain decoding work has shown the potential of Vision Transformers (ViTs) to learn mappings from neural representation to visual features. Here we present CortexDecoder, a brain decoding framework mapping fMRI flatmaps to images viewed by subjects. Flatmaps are neural responses projected onto cortical surfaces and then unfolded into 2D images. We demonstrate that (1) flatmaps preserve sufficient signal to achieve high performance in brain decoding; raw fMRI data has noisy signals and high-dimensionality which makes decoding a difficult task. (2) Reshaping raw fMRI data into flatmaps allows for clean patch-based tokenization. This allows us to harness state-of-the-art Vision Transformers to learn meaningful neural representations and preserve spatial encoding. To do this, we pass flatmap patches into a pretrained CortexMAE encoder to obtain brain tokens. A lightweight cross-attention bridge then learns to map these brain tokens to image features. Our cross-attention bridge is split into two modules; the semantic module for predicting CLIP embeddings and the lower-level module to generate low-level image reconstruction. Finally, a fine-tuned SDXL (Stable Diffusion XL) takes this low-level reconstruction image as the initial image. Conditioned by the CLIP semantic embedding, the SDXL produces the final reconstruction image. In single-subject evaluations, CortexDecoder achieves performance comparable to state-of-the-art brain decoding methods, demonstrating the effectiveness of flatmap based neural representations for visual reconstruction.
Related
- FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps
- Infection-related images reduce cortical edge metastability: a specification-curve analysis of a block-design fMRI study
- Association between red blood cell count and cortical functional connectivity in type 2 diabetes mellitus patients residing at high altitude: resting-state fMRI study in the Tibetan Plateau
- Intrinsic Activity Reductions and Cerebello-Cortical Diaschisis-Like Mechanism in Friedreich Ataxia: An fALFF Resting-State fMRI Study.
- Predicting cortical signal mixing in coarse-resolution fMRI: a forward-operator analysis of surface mapping and the conditional benefit of correction
BibTeX
@misc{imototar2026cortexdecoder,
title = {{CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps}},
author = {T. Imoto-Tar and M. Fung and A. Gurha},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.09.757766},
url = {https://www.biorxiv.org/content/10.64898/2026.10.09.757766v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps
AU - T. Imoto-Tar
AU - M. Fung
AU - A. Gurha
PY - 2026
DA - 2026/10/09
PB - bioRxiv
DO - 10.64898/2026.10.09.757766
UR - https://www.biorxiv.org/content/10.64898/2026.10.09.757766v1
AB - Recent advances in brain decoding work has shown the potential of Vision Transformers (ViTs) to learn mappings from neural representation to visual features. Here we present CortexDecoder, a brain decoding framework mapping fMRI flatmaps to images viewed by subjects. Flatmaps are neural responses projected onto cortical surfaces and then unfolded into 2D images. We demonstrate that (1) flatmaps preserve sufficient signal to achieve high performance in brain decoding; raw fMRI data has noisy signals and high-dimensionality which makes decoding a difficult task. (2) Reshaping raw fMRI data into flatmaps allows for clean patch-based tokenization. This allows us to harness state-of-the-art Vision Transformers to learn meaningful neural representations and preserve spatial encoding. To do this, we pass flatmap patches into a pretrained CortexMAE encoder to obtain brain tokens. A lightweight cross-attention bridge then learns to map these brain tokens to image features. Our cross-attention bridge is split into two modules; the semantic module for predicting CLIP embeddings and the lower-level module to generate low-level image reconstruction. Finally, a fine-tuned SDXL (Stable Diffusion XL) takes this low-level reconstruction image as the initial image. Conditioned by the CLIP semantic embedding, the SDXL produces the final reconstruction image. In single-subject evaluations, CortexDecoder achieves performance comparable to state-of-the-art brain decoding methods, demonstrating the effectiveness of flatmap based neural representations for visual reconstruction.
ER -