Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies
Abstract
Multimodal large language models (MLLMs) offer scalable annotations of social and affective content, but their temporal agreement with human judgments and neural predictive value require further evaluation. We compared diffusion-based and autoregressive MLLMs with continuous human Theory of Mind (ToM) and affective valence ratings across three short cartoon movies. Behavioral analyses assessed temporally aligned Pearson correlations, rating distributions, and the effects of preceding scene descriptions. Ratings from each source were separately used in subject-specific regression models to predict regional fMRI responses. Model ratings captured aspects of human temporal variation despite systematic distributional differences, with generally stronger alignment for valence than ToM. Preceding context improved alignment in some conditions, depending on the model and movie. In \emph{The Present}, more regions showed significant encoding for MLLM ratings than for human ratings, whereas effects were more limited in the other movies. Spatial overlap across rating sources occurred mainly among MLLMs; overlap between human and model ratings was limited and involved primarily visual, dorsal-attention, and posterior default-mode regions. These findings support MLLMs as scalable tools for generating brain-predictive annotations of social and emotional content, while highlighting systematic rating differences and model- and stimulus-dependent alignment with human judgments.
Related
- Multi-region social-emotional network underlying the effects of spine pinching therapy on empathy-like behavior in a rat model of autism spectrum disorder
- Understanding the role of the hippocampus in representing multimodal stimuli
- Grounding meaning in social and emotional experience: Challenges and opportunities
- Associations Between Neural Responses to Social Fairness and Financial Exploitation Vulnerability Vary With Age and Emotional Intelligence
- Exploring neural correlates of emotional and social listening with high-density diffuse optical tomography in the auditory cortex
BibTeX
@misc{huang2026brainaligned,
title = {{Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies}},
author = {Yiran Huang and Zeyong Zhang and Bharat Biswal and Xin Di and Mengjia Xu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.22706},
eprint = {2610.22706},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.22706},
}
RIS
TY - UNPB
TI - Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies
AU - Yiran Huang
AU - Zeyong Zhang
AU - Bharat Biswal
AU - Xin Di
AU - Mengjia Xu
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.22706
UR - https://arxiv.org/abs/2610.22706
AB - Multimodal large language models (MLLMs) offer scalable annotations of social and affective content, but their temporal agreement with human judgments and neural predictive value require further evaluation. We compared diffusion-based and autoregressive MLLMs with continuous human Theory of Mind (ToM) and affective valence ratings across three short cartoon movies. Behavioral analyses assessed temporally aligned Pearson correlations, rating distributions, and the effects of preceding scene descriptions. Ratings from each source were separately used in subject-specific regression models to predict regional fMRI responses. Model ratings captured aspects of human temporal variation despite systematic distributional differences, with generally stronger alignment for valence than ToM. Preceding context improved alignment in some conditions, depending on the model and movie. In \emph{The Present}, more regions showed significant encoding for MLLM ratings than for human ratings, whereas effects were more limited in the other movies. Spatial overlap across rating sources occurred mainly among MLLMs; overlap between human and model ratings was limited and involved primarily visual, dorsal-attention, and posterior default-mode regions. These findings support MLLMs as scalable tools for generating brain-predictive annotations of social and emotional content, while highlighting systematic rating differences and model- and stimulus-dependent alignment with human judgments.
ER -