EmoMind: Decoding Affective Captions from Human Brain fMRI
Abstract
Decoding visual experience from brain activity has advanced substantially, but current brain-to-text systems largely recover semantic content while discarding affect. Additionally, language models can generate emotional text when prompted with categorical labels, but such labels collapse rich inter-subject variability into coarse discrete bins. We present EmoMind, the first end-to-end pipeline for decoding affective captions directly from fMRI signals. EmoMind first retrieves a semantically grounded neutral scene description from brain-decoded visual features, then rewrites it using a continuous 34-dimensional emotion vector decoded from the same fMRI recording. To control the balance between content preservation and affective expression, we train the rewriter with classifier-free guidance against an identity-preserving null branch, enabling smooth interpolation between semantic fidelity and affective expressivity. We evaluate affective caption generation with a three-axis validation framework spanning subject-specificity, structural geometry, and causal control. We further augment this framework with a synthetic-brain substitution test that probes robustness to the measurement apparatus, and we benchmark each axis against GPT-4 prompted with brain-decoded top-5 emotion labels as a strong discrete baseline. Across two independent emotion fMRI datasets, EmoMind significantly outperforms label-prompted GPT-4 on all three axes, with the largest gains on metrics that require person-specific affective structure rather than population-level emotion aggregation. These results establish continuous brain-decoded affect as a viable control signal for individualized affective caption generation and open new directions for studying individual affective brain organisation.
Related
- Noninvasive decoding of typed sentences from human brain activity
- MBB-AVR: a mind-brain-body dataset with dynamic affective states in immersive virtual reality
- Affective representations and brain-body dynamics during emotional video viewing
- Mentalizing from self and friend perspectives: Relations between inter-brain neural similarity, affective similarity and social anxiety symptoms.
- Toward human-centred resilience quantification: The role of eco-affective state variables
BibTeX
@misc{mohammed2026emomind,
title = {{EmoMind: Decoding Affective Captions from Human Brain fMRI}},
author = {Bilal A. Mohammed and Lin Gu and Ruogu Fang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.16739},
eprint = {2605.16739},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.16739},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - EmoMind: Decoding Affective Captions from Human Brain fMRI
AU - Bilal A. Mohammed
AU - Lin Gu
AU - Ruogu Fang
PY - 2026
DA - 2026/09/07
PB - arXiv
DO - 10.48550/arxiv.2605.16739
UR - https://arxiv.org/abs/2605.16739
AB - Decoding visual experience from brain activity has advanced substantially, but current brain-to-text systems largely recover semantic content while discarding affect. Additionally, language models can generate emotional text when prompted with categorical labels, but such labels collapse rich inter-subject variability into coarse discrete bins. We present EmoMind, the first end-to-end pipeline for decoding affective captions directly from fMRI signals. EmoMind first retrieves a semantically grounded neutral scene description from brain-decoded visual features, then rewrites it using a continuous 34-dimensional emotion vector decoded from the same fMRI recording. To control the balance between content preservation and affective expression, we train the rewriter with classifier-free guidance against an identity-preserving null branch, enabling smooth interpolation between semantic fidelity and affective expressivity. We evaluate affective caption generation with a three-axis validation framework spanning subject-specificity, structural geometry, and causal control. We further augment this framework with a synthetic-brain substitution test that probes robustness to the measurement apparatus, and we benchmark each axis against GPT-4 prompted with brain-decoded top-5 emotion labels as a strong discrete baseline. Across two independent emotion fMRI datasets, EmoMind significantly outperforms label-prompted GPT-4 on all three axes, with the largest gains on metrics that require person-specific affective structure rather than population-level emotion aggregation. These results establish continuous brain-decoded affect as a viable control signal for individualized affective caption generation and open new directions for studying individual affective brain organisation.
ER -