FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps
Abstract
Can the spatial organization of cortical activity help transfer frozen image representations to fMRI prediction? We introduce FlatClip, which renders cortical activity as anatomy-linked flatmaps and extracts features with a frozen SigLIP2 encoder. Only a lightweight downstream probe is trained, without fMRI-specific encoder pretraining or fine-tuning. Across four resting-state benchmarks, FlatClip outperforms the evaluated ROI-level baselines on HCP and ADNI, but is weaker on PPMI and below the strongest voxel-level models overall. On NSD visual-fMRI decoding, restricting inputs to visual or NSD-provided task-active cortex improves recognition over whole-cortex inputs. Spatial perturbations reduce performance with both retrained and fixed readouts, and anatomy-linked arrangements outperform cortical-pixel permutations across three colormaps. Together, these results demonstrate a practical route for reusing image-pretrained features in fMRI and highlight cortical arrangement and input-region selection as important design choices. Code is available at https://github.com/OneMore1/FlatClip.
Related
- CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps
- Representation learning of human cortical folding to reveal long lasting neurodevelopmental signatures
- AREION: A Foundation Model for Atlas-Agnostic Representation Learning of Brain Connectome
- scRep: A Latent-Space Self-Distilled Foundation Model for Single-Cell Representation Learning
- Action features dominate cortical representation during natural vision
BibTeX
@misc{wang2026flatclip,
title = {{FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps}},
author = {Mo Wang and Wenhao Ye and Zihan Ning and Jiayu Zuo and Junfeng Xia and Hongkai Wen and Quanying Liu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.31204},
eprint = {2609.31204},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.31204},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps
AU - Mo Wang
AU - Wenhao Ye
AU - Zihan Ning
AU - Jiayu Zuo
AU - Junfeng Xia
AU - Hongkai Wen
AU - Quanying Liu
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2609.31204
UR - https://arxiv.org/abs/2609.31204
AB - Can the spatial organization of cortical activity help transfer frozen image representations to fMRI prediction? We introduce FlatClip, which renders cortical activity as anatomy-linked flatmaps and extracts features with a frozen SigLIP2 encoder. Only a lightweight downstream probe is trained, without fMRI-specific encoder pretraining or fine-tuning. Across four resting-state benchmarks, FlatClip outperforms the evaluated ROI-level baselines on HCP and ADNI, but is weaker on PPMI and below the strongest voxel-level models overall. On NSD visual-fMRI decoding, restricting inputs to visual or NSD-provided task-active cortex improves recognition over whole-cortex inputs. Spatial perturbations reduce performance with both retrained and fixed readouts, and anatomy-linked arrangements outperform cortical-pixel permutations across three colormaps. Together, these results demonstrate a practical route for reusing image-pretrained features in fMRI and highlight cortical arrangement and input-region selection as important design choices. Code is available at https://github.com/OneMore1/FlatClip.
ER -