Unimodal Augmentation-based SSL Objectives Induce Cross-Modal Alignment
Abstract
Recent works have observed implicit cross-modal alignment between independently trained unimodal models. However, the mechanisms underlying the emergence of this phenomenon remain unresolved. In this paper, we demonstrate that unimodal augmentation-based self-supervised objectives implicitly induce cross-modal alignment. We show that low unimodal losses force high cross-modal alignment, whereas a high unimodal loss prevents such alignment. We theoretically connect the widely used InfoNCE loss to local CKA, the popular measure of cross-modal alignment. We further provide extensive empirical evidence supporting this claim, both in controlled settings and across 53 unconstrained frozen text, vision, and audio unimodal encoders. In both cases, unimodal augmentation-based objectives control cross-modal alignment. Overall, our results provide a theoretical and empirical foundation for understanding the emergence of implicit cross-modal alignment by establishing a direct connection between unimodal training objectives and cross-modal structure.
Related
- A target-domain augmentation and class-wise alignment framework for few-shot cross-condition bearing fault diagnosis
- Conguences for Odd Unimodal Sequences and a Problem of Bringmann And Lovejoy
- Existence of the Lyapunov exponent for S-unimodal maps
- Target alignment, dilution and forecast selection when cross-sectional forecasts share a common target
- A linear dual-field finite element method for incompressible MHD, part I: cross-helicity and Alfvénic alignment in periodic domains
BibTeX
@misc{mashiah2026unimodal,
title = {{Unimodal Augmentation-based SSL Objectives Induce Cross-Modal Alignment}},
author = {Zohar Mashiah and Nir Ben-Ari and Ronen Talmon and Uri Shaham},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.02917},
eprint = {2611.02917},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.02917},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Unimodal Augmentation-based SSL Objectives Induce Cross-Modal Alignment
AU - Zohar Mashiah
AU - Nir Ben-Ari
AU - Ronen Talmon
AU - Uri Shaham
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.02917
UR - https://arxiv.org/abs/2611.02917
AB - Recent works have observed implicit cross-modal alignment between independently trained unimodal models. However, the mechanisms underlying the emergence of this phenomenon remain unresolved. In this paper, we demonstrate that unimodal augmentation-based self-supervised objectives implicitly induce cross-modal alignment. We show that low unimodal losses force high cross-modal alignment, whereas a high unimodal loss prevents such alignment. We theoretically connect the widely used InfoNCE loss to local CKA, the popular measure of cross-modal alignment. We further provide extensive empirical evidence supporting this claim, both in controlled settings and across 53 unconstrained frozen text, vision, and audio unimodal encoders. In both cases, unimodal augmentation-based objectives control cross-modal alignment. Overall, our results provide a theoretical and empirical foundation for understanding the emergence of implicit cross-modal alignment by establishing a direct connection between unimodal training objectives and cross-modal structure.
ER -