Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI
Abstract
Decoding sensory-evoked functional magnetic resonance imaging (fMRI) data is a central analytical tool in cognitive neuroscience. Here we ask whether self-supervised learning (SSL), which has been scarcely explored in this domain, can yield representations that support strong downstream performance. To do so, we adapt a joint embedding predictive architecture (JEPA) to train participant-specific encoders on brain data alone. The encoders never see the stimulus images or any representation derived from them. Applied to the Natural Scenes Dataset (NSD), we evaluate our learned brain embeddings on the downstream tasks of brain captioning and image reconstruction, as well as cross-participant generalization. Our self-supervised approach achieves competitive performance with parameter-rich supervised pipelines that are trained end-to-end with access to image and language data. We further find that encoders trained independently for different participants can be aligned fully unsupervised, using neither images nor stimuli shared between them. After alignment, a neural decoding head trained on one participant can be applied to another without fine-tuning and retains most of its within-subject performance on image reconstruction and brain captioning. By removing the need for images and shared stimuli, our approach is a step towards pooling fMRI data across participants and, eventually, across datasets.
Related
- Perceptual similarity representations beyond the sensory cortex
- Nonlinear sensory integration drives configural olfactory representations
- Goal-directed action selection relies on conjunctive representations that bind goals, actions and expected sensory outcomes
- Highly resolved representations of dynamic haptic inputs in non-primary sensory cortical neuron representations
- Odor representations are structured by heterogeneous tuning and concentration-dependence of functionally distinct olfactory sensory neuron subpopulations.
BibTeX
@misc{jassim2026latent,
title = {{Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI}},
author = {Serwan Jassim and Victoria Bosch and Daniel Anthes and Elia Bruni and Tim Kietzmann},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.40280},
eprint = {2610.40280},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.40280},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI
AU - Serwan Jassim
AU - Victoria Bosch
AU - Daniel Anthes
AU - Elia Bruni
AU - Tim Kietzmann
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.40280
UR - https://arxiv.org/abs/2610.40280
AB - Decoding sensory-evoked functional magnetic resonance imaging (fMRI) data is a central analytical tool in cognitive neuroscience. Here we ask whether self-supervised learning (SSL), which has been scarcely explored in this domain, can yield representations that support strong downstream performance. To do so, we adapt a joint embedding predictive architecture (JEPA) to train participant-specific encoders on brain data alone. The encoders never see the stimulus images or any representation derived from them. Applied to the Natural Scenes Dataset (NSD), we evaluate our learned brain embeddings on the downstream tasks of brain captioning and image reconstruction, as well as cross-participant generalization. Our self-supervised approach achieves competitive performance with parameter-rich supervised pipelines that are trained end-to-end with access to image and language data. We further find that encoders trained independently for different participants can be aligned fully unsupervised, using neither images nor stimuli shared between them. After alignment, a neural decoding head trained on one participant can be applied to another without fine-tuning and retains most of its within-subject performance on image reconstruction and brain captioning. By removing the need for images and shared stimuli, our approach is a step towards pooling fMRI data across participants and, eventually, across datasets.
ER -