HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning
Abstract
Spatial transcriptomics maps gene expression across tissue sections and, when integrated with whole slide images, jointly captures both morphological architecture and molecular activity to provide a rich, multimodal view of tissue biology. ML approaches combining spatial transcriptomics and WSIs typically learn a mapping from morphological features to molecular profiles via contrastive alignment or supervised fine-tuning of pathology foundation models. In both cases, transcriptomics serves as a training signal rather than a true input modality, leaving inference fundamentally image-driven. We propose an early-fusion vision transformer architecture in which transcript tokens are constructed at arbitrary spatial resolution, merged with image patch tokens, and jointly processed through a shared transformer stack for deep cross-modal interaction. Starting from a pretrained vision-only pathology foundation model, we train HistoTx by continuing self-supervised pretraining on paired image and spatial transcriptomics data and demonstrate that: 1) at inference with image only, performance remains on par with the baseline on most tasks while improving on tasks that inherently benefit from molecular knowledge, such as gene expression prediction; 2) when both modalities are available at inference, jointly providing image and transcript inputs outperforms either modality alone.
Related
- Cellular-Resolution Spatial Transcriptomics Reveals Laminar VSMC Phenotypic Remodeling and a Hypoxic Medial Core in Human Thoracic Aortic Dissection
- SpaceBlocks: an expert-supervised workflow for cohort-level spatial transcriptomics analysis
- HESTIA: Scalable Multimodal Integration of Histology and High-Resolution Spatial Transcriptomics for Robust Spatial Domain Identification
- ASTRA enables zero-shot super-resolution of spatial transcriptomics across capture geometries
- Imaging-based spatial isoform transcriptomics resolves transcript diversity at cellular and subcellular resolution
BibTeX
@misc{driessen2026histotx,
title = {{HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning}},
author = {Alice Driessen and Emil Malmsten and Steffen Wolf and Bernd Illing and Isinsu Katircioglu and Adriano Martinelli and Rolf Stahel and Solange Peters and Enriqueta Felip and Sanjay Popat and Krisztian Homicsko and Raphael Gottardo and Marianna Rapsomaniki and Fei Tang},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.02681},
eprint = {2611.02681},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.02681},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning
AU - Alice Driessen
AU - Emil Malmsten
AU - Steffen Wolf
AU - Bernd Illing
AU - Isinsu Katircioglu
AU - Adriano Martinelli
AU - Rolf Stahel
AU - Solange Peters
AU - Enriqueta Felip
AU - Sanjay Popat
AU - Krisztian Homicsko
AU - Raphael Gottardo
AU - Marianna Rapsomaniki
AU - Fei Tang
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.02681
UR - https://arxiv.org/abs/2611.02681
AB - Spatial transcriptomics maps gene expression across tissue sections and, when integrated with whole slide images, jointly captures both morphological architecture and molecular activity to provide a rich, multimodal view of tissue biology. ML approaches combining spatial transcriptomics and WSIs typically learn a mapping from morphological features to molecular profiles via contrastive alignment or supervised fine-tuning of pathology foundation models. In both cases, transcriptomics serves as a training signal rather than a true input modality, leaving inference fundamentally image-driven. We propose an early-fusion vision transformer architecture in which transcript tokens are constructed at arbitrary spatial resolution, merged with image patch tokens, and jointly processed through a shared transformer stack for deep cross-modal interaction. Starting from a pretrained vision-only pathology foundation model, we train HistoTx by continuing self-supervised pretraining on paired image and spatial transcriptomics data and demonstrate that: 1) at inference with image only, performance remains on par with the baseline on most tasks while improving on tasks that inherently benefit from molecular knowledge, such as gene expression prediction; 2) when both modalities are available at inference, jointly providing image and transcript inputs outperforms either modality alone.
ER -