Human-like meaning maps from single-prompt VLM ratings of local scene meaning
- 1McGill University
Abstract
Visual attention is shaped both by low-level perceptual features and by the semantic properties of scene regions. However, compared to the salience of low-level image features, the characterization of image properties that lead to scene understanding has been more difficult. The semantic properties of scene regions have been operationalized via meaning maps, proposed by Henderson and Hayes (2017), by having human observers rate the informativeness of scene patches. However, this rating procedure is labor-intensive and limits the method's versatility. An automated alternative, the DeepMeaning model, reduces rating costs but still requires human training data from a similar class of images, limiting generalizability. Here, we tested whether instruction-following vision-language models (VLMs) can generate human-like ratings in a zero-shot manner: using the original human-rating instructions as a single prompt without any task-specific training. We obtained meaningfulness ratings from both open-weight and proprietary VLMs for scene patches across three meaning-map datasets, and compared the resulting zero-shot AI meaning maps (AIMMs) to human meaning maps and maps obtained from the DeepMeaning model. Zero-shot AIMMs closely approximate human meaning maps, and maps from the task-specific DeepMeaning model, showing that VLMs can replicate human semantic judgments without any training. Zero-shot AIMMs demonstrate utility and validity, matching human meaning maps in their ability to predict human fixations and reproducing the same pattern of relative performance against other gaze-prediction models. We provide open-source code that allows researchers to use this zero-shot approach to quickly and flexibly explore meaning maps across spatial scales, prompts, and scene categories.
Related
- Large language models poorly capture how word order shapes representations of sentence meaning in the human brain.
- Grid-like and distance codes for representing word meaning in the human brain
- Choice Begins Before Choice: Meaning-Shaped Accessibility, Live Possibility, and Effective Agency in Human-AI Systems (Full Paper v1.0, strong-claim synthesis)
- Beyond reproduction: Reclaiming the full meaning of sexual and reproductive health
- Grounding meaning in social and emotional experience: Challenges and opportunities
BibTeX
@misc{jurewicz2026humanlike,
title = {{Human-like meaning maps from single-prompt VLM ratings of local scene meaning}},
author = {K. Jurewicz and Y.-E. Berreby and S. Krishna},
howpublished = {bioRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.30.748100},
url = {https://www.biorxiv.org/content/10.64898/2026.08.30.748100v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Human-like meaning maps from single-prompt VLM ratings of local scene meaning
AU - K. Jurewicz
AU - Y.-E. Berreby
AU - S. Krishna
PY - 2026
DA - 2026/08/30
PB - bioRxiv
DO - 10.64898/2026.08.30.748100
UR - https://www.biorxiv.org/content/10.64898/2026.08.30.748100v1
AB - Visual attention is shaped both by low-level perceptual features and by the semantic properties of scene regions. However, compared to the salience of low-level image features, the characterization of image properties that lead to scene understanding has been more difficult. The semantic properties of scene regions have been operationalized via meaning maps, proposed by Henderson and Hayes (2017), by having human observers rate the informativeness of scene patches. However, this rating procedure is labor-intensive and limits the method's versatility. An automated alternative, the DeepMeaning model, reduces rating costs but still requires human training data from a similar class of images, limiting generalizability. Here, we tested whether instruction-following vision-language models (VLMs) can generate human-like ratings in a zero-shot manner: using the original human-rating instructions as a single prompt without any task-specific training. We obtained meaningfulness ratings from both open-weight and proprietary VLMs for scene patches across three meaning-map datasets, and compared the resulting zero-shot AI meaning maps (AIMMs) to human meaning maps and maps obtained from the DeepMeaning model. Zero-shot AIMMs closely approximate human meaning maps, and maps from the task-specific DeepMeaning model, showing that VLMs can replicate human semantic judgments without any training. Zero-shot AIMMs demonstrate utility and validity, matching human meaning maps in their ability to predict human fixations and reproducing the same pattern of relative performance against other gaze-prediction models. We provide open-source code that allows researchers to use this zero-shot approach to quickly and flexibly explore meaning maps across spatial scales, prompts, and scene categories.
ER -