VINTER: a generative vision-language model for annotating and interrogating spatial tissues
- 1Institute of Medical Science, University of Tokyo
Abstract
Spatial transcriptomics links gene expression to tissue structure, yet current models learn embeddings whose meaning is assigned afterwards or read histology without measured expression. Tissue decisions are therefore rarely made from morphology, genes and neighbourhood together, or examined through them. Here we introduce VINTER, a generative vision–language model aligned on 1.45 million histology–caption pairs that reads histology, measured gene expression and spatial neighbourhood as one sequence. It returns the probability of every tissue class, and because genes enter as text, its molecular evidence can be edited with the image held fixed. Reading all three sources together, VINTER annotated tissue more accurately than spatial-domain methods and a spatial foundation model, and transferred across cohorts without target labels. Editing only the neighbourhood resolved the follicular B-cell and plasma-cell programmes behind its reading of tertiary lymphoid structure (TLS) maturation in lung and kidney cancer. Its class probabilities exposed what single labels conceal in triple-negative breast cancer: immune heterogeneity within stroma and cancer hidden beneath benign labels. Without retraining, the same model transferred to Visium HD and single-cell Xenium data and resolved follicular architecture within colorectal cancer TLS. VINTER thus turns spatial annotation into an interrogable reading of tissue across platforms and resolutions.
Related
- Temporal and spatial atlas of eosinophil specialization across tissues
- Decoder-FFPE-seq enables sensitive, genome-wide spatial transcriptomics of archival tissues at single-cell resolution
- Symbiont spatial organisation is dynamically regulated within cnidarian host tissues
- Generative design of bacteriophages with genome language models
- Distinct Spatial Immune Architectures in Tumor and Tumor-Adjacent Tissues of Early-Stage Non-Small Cell Lung Cancer
BibTeX
@misc{wu2026vinter,
title = {{VINTER: a generative vision-language model for annotating and interrogating spatial tissues}},
author = {J. Wu and W. Xu and Y. Zhuang and Y. Zhang and M. d. J. Loza Lopez and y. yang and K. Nakai},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.01.752251},
url = {https://www.biorxiv.org/content/10.64898/2026.10.01.752251v1},
note = {Public domain},
}
RIS
TY - UNPB
TI - VINTER: a generative vision-language model for annotating and interrogating spatial tissues
AU - J. Wu
AU - W. Xu
AU - Y. Zhuang
AU - Y. Zhang
AU - M. d. J. Loza Lopez
AU - y. yang
AU - K. Nakai
PY - 2026
DA - 2026/10/01
PB - bioRxiv
DO - 10.64898/2026.10.01.752251
UR - https://www.biorxiv.org/content/10.64898/2026.10.01.752251v1
AB - Spatial transcriptomics links gene expression to tissue structure, yet current models learn embeddings whose meaning is assigned afterwards or read histology without measured expression. Tissue decisions are therefore rarely made from morphology, genes and neighbourhood together, or examined through them. Here we introduce VINTER, a generative vision–language model aligned on 1.45 million histology–caption pairs that reads histology, measured gene expression and spatial neighbourhood as one sequence. It returns the probability of every tissue class, and because genes enter as text, its molecular evidence can be edited with the image held fixed. Reading all three sources together, VINTER annotated tissue more accurately than spatial-domain methods and a spatial foundation model, and transferred across cohorts without target labels. Editing only the neighbourhood resolved the follicular B-cell and plasma-cell programmes behind its reading of tertiary lymphoid structure (TLS) maturation in lung and kidney cancer. Its class probabilities exposed what single labels conceal in triple-negative breast cancer: immune heterogeneity within stroma and cancer hidden beneath benign labels. Without retraining, the same model transferred to Visium HD and single-cell Xenium data and resolved follicular architecture within colorectal cancer TLS. VINTER thus turns spatial annotation into an interrogable reading of tissue across platforms and resolutions.
ER -