Zero-shot evaluations expose structured generalization limits in predictive models of the brain
- 1Georgia Institute of Technology
Abstract
Artificial neural networks (ANNs) can predict neural responses to natural images with remarkable accuracy, making them promising models of human vision. However, prediction scores alone reveal little about what computations these models have captured or how well they generalize beyond the images on which they are trained. Here, we introduce a principled zero-shot model evaluation framework in which ANN-based encoding models are frozen before being tested on not just held out images but entirely new datasets. Our strongest tests repurpose decades of cognitive neuroscience experiments as diagnostic benchmarks for identifying which computations current models capture and where they fail. To implement this framework we built encoding models of human category-selective regions (FFA, PPA, and EBA) using several pretrained ANNs and evaluated these fixed models across 7 independent fMRI datasets and 31 cognitive experiments from 10 published studies. This framework revealed three broad patterns. First, zero-shot evaluations exposed systematic limits to model predictivity that were largely hidden by standard within-dataset cross-validation. These limits were structured varying across brain regions and model classes and depending strongly on the similarity between the images used to build and test the models. Second, current models reproduced some, but not all, classical findings from cognitive neuroscience, providing a diagnosis of the computations that current models have yet to capture. Finally, models that performed well on prediction tests also tended to succeed at reproducing cognitive neuroscience findings, suggesting that prediction and explanation are closely linked. Unlike prediction scores however, cognitive tests help diagnose where and why models fail. Together, the zero-shot evaluation framework provides a scalable and unified approach for evaluating brain models across prediction and cognitive neuroscience tests, which can help us better understand what current models capture and what still remains to be explained.
Related
- Leakage-controlled benchmarking reveals generalization limits of deep learning for protein-ligand binding affinity prediction
- Direct input variables: rethinking the limits of the extended mind hypothesis within predictive processing.
- Predictive Multiplicity in Cell-Fate Assignment: Label-Free Rashomon Sets and the Limits of Per-Cell Certification
- When perfect is a warning: explanations expose shortcuts that accuracy hides in cancer gene-expression classifiers
- German dairy cows expose heterogenous bovine meat and milk factor DNA repertoires in milk and reveal Rep expression in mammary tissue
BibTeX
@misc{wang2026zeroshot,
title = {{Zero-shot evaluations expose structured generalization limits in predictive models of the brain}},
author = {R. Wang and M. Deb and A. Abate and A. Dipani and K. R. Dudipala and S. Chillarege and K. Ravikanti and Y. Li and H. Al-Tahan and R. Koushik and E. Mieczkowski and H. Fung and N. Kanwisher and N. A. Ratan Murty},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.15.751562},
url = {https://www.biorxiv.org/content/10.64898/2026.09.15.751562v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Zero-shot evaluations expose structured generalization limits in predictive models of the brain
AU - R. Wang
AU - M. Deb
AU - A. Abate
AU - A. Dipani
AU - K. R. Dudipala
AU - S. Chillarege
AU - K. Ravikanti
AU - Y. Li
AU - H. Al-Tahan
AU - R. Koushik
AU - E. Mieczkowski
AU - H. Fung
AU - N. Kanwisher
AU - N. A. Ratan Murty
PY - 2026
DA - 2026/09/15
PB - bioRxiv
DO - 10.64898/2026.09.15.751562
UR - https://www.biorxiv.org/content/10.64898/2026.09.15.751562v1
AB - Artificial neural networks (ANNs) can predict neural responses to natural images with remarkable accuracy, making them promising models of human vision. However, prediction scores alone reveal little about what computations these models have captured or how well they generalize beyond the images on which they are trained. Here, we introduce a principled zero-shot model evaluation framework in which ANN-based encoding models are frozen before being tested on not just held out images but entirely new datasets. Our strongest tests repurpose decades of cognitive neuroscience experiments as diagnostic benchmarks for identifying which computations current models capture and where they fail. To implement this framework we built encoding models of human category-selective regions (FFA, PPA, and EBA) using several pretrained ANNs and evaluated these fixed models across 7 independent fMRI datasets and 31 cognitive experiments from 10 published studies. This framework revealed three broad patterns. First, zero-shot evaluations exposed systematic limits to model predictivity that were largely hidden by standard within-dataset cross-validation. These limits were structured varying across brain regions and model classes and depending strongly on the similarity between the images used to build and test the models. Second, current models reproduced some, but not all, classical findings from cognitive neuroscience, providing a diagnosis of the computations that current models have yet to capture. Finally, models that performed well on prediction tests also tended to succeed at reproducing cognitive neuroscience findings, suggesting that prediction and explanation are closely linked. Unlike prediction scores however, cognitive tests help diagnose where and why models fail. Together, the zero-shot evaluation framework provides a scalable and unified approach for evaluating brain models across prediction and cognitive neuroscience tests, which can help us better understand what current models capture and what still remains to be explained.
ER -