Vision-language encoding models reveal an image-computable food-quality dimension in human occipitotemporal cortex
- 1Maastricht University
Abstract
Perceived calorie content contributes to neural representational structure in human ventral visual cortex, yet it remains unclear whether this reflects an abstract nutritional signal or whether perceived calorie is largely recoverable from the visual-semantic structure of the food image itself. In 25 female participants who passively viewed 96 food images during functional MRI, we decomposed perceived calorie ratings into a component predicted from CLIP (Contrastive Language-Image Pretraining) image embeddings, a vision-language model that captures high-level visual-semantic image structure, and a residual component not captured by this CLIP-based prediction. We then tested their respective contributions to neural prediction using cross-validated banded ridge encoding models. The CLIP-predictable component organized foods along a processedness and naturalness dimension, separating raw single-ingredient foods from prepared and energy-dense foods. Adding this component to a visual-semantic baseline improved neural prediction progressively along the ventral visual hierarchy, with the strongest relative contribution in higher-level ventral temporal cortex. These findings indicate that calorie-related encoding in ventral visual cortex is carried mainly by a shared food-quality axis indexing processedness, naturalness, and perceived healthiness, a substantial part of which is recoverable from image-computable visual-semantic structure, rather than providing evidence for an isolated abstract representation of caloric magnitude. Because these results derive from a reanalysis of 25 female participants viewing a fixed set of 96 images, generalization to broader populations and larger, more varied stimulus sets remains to be established.
Related
- Human see, human do? Viewing tool pictures evokes tool-use action information in hand-selective occipitotemporal cortex
- Comparing serial and parallel language input: Shared neural representations with distinct encoding strengths
- LigRNA: High-accuracy prediction of RNA-ligand binding sites via dynamic chemical complementarity and language model encoding
- Principles of coarse-scale functional organization in occipitotemporal cortex
- Experiences administering the EuroQol five-dimension five-level (EQ-5D-5L) for valuation of health-related quality of life estimates in Kenya: An embedded qualitative study
BibTeX
@misc{marrazzo2026visionlanguage,
title = {{Vision-language encoding models reveal an image-computable food-quality dimension in human occipitotemporal cortex}},
author = {G. Marrazzo and A. Roefs and L. Pimpini},
howpublished = {bioRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.25.747006},
url = {https://www.biorxiv.org/content/10.64898/2026.08.25.747006v2},
note = {CC BY},
}
RIS
TY - UNPB
TI - Vision-language encoding models reveal an image-computable food-quality dimension in human occipitotemporal cortex
AU - G. Marrazzo
AU - A. Roefs
AU - L. Pimpini
PY - 2026
DA - 2026/08/25
PB - bioRxiv
DO - 10.64898/2026.08.25.747006
UR - https://www.biorxiv.org/content/10.64898/2026.08.25.747006v2
AB - Perceived calorie content contributes to neural representational structure in human ventral visual cortex, yet it remains unclear whether this reflects an abstract nutritional signal or whether perceived calorie is largely recoverable from the visual-semantic structure of the food image itself. In 25 female participants who passively viewed 96 food images during functional MRI, we decomposed perceived calorie ratings into a component predicted from CLIP (Contrastive Language-Image Pretraining) image embeddings, a vision-language model that captures high-level visual-semantic image structure, and a residual component not captured by this CLIP-based prediction. We then tested their respective contributions to neural prediction using cross-validated banded ridge encoding models. The CLIP-predictable component organized foods along a processedness and naturalness dimension, separating raw single-ingredient foods from prepared and energy-dense foods. Adding this component to a visual-semantic baseline improved neural prediction progressively along the ventral visual hierarchy, with the strongest relative contribution in higher-level ventral temporal cortex. These findings indicate that calorie-related encoding in ventral visual cortex is carried mainly by a shared food-quality axis indexing processedness, naturalness, and perceived healthiness, a substantial part of which is recoverable from image-computable visual-semantic structure, rather than providing evidence for an isolated abstract representation of caloric magnitude. Because these results derive from a reanalysis of 25 female participants viewing a fixed set of 96 images, generalization to broader populations and larger, more varied stimulus sets remains to be established.
ER -