Shared Gaussianization: What Gaussian Regularizers Certify About Contrastive Learning, and What They Miss
Abstract
What can a distribution-matching regularizer such as SIGReg in LeJEPA certify about contrastive learning? We study shared Gaussianization (SG), a characteristic-function Gaussianity test on the average of two normalized views, scaled by an independent χd radius. Because disagreeing views shorten the average, one test detects both misalignment and non-uniformity. SG vanishes exactly at the aligned, uniform minimizers of population InfoNCE, and under equal marginals it bounds the InfoNCE excess by 4· 33/4β times the square root of the SG loss, plus a term linear in the loss. The square-root rate and this dimension-free constant are sharp, and no squared mean-embedding distance on view pairs achieves a faster rate. With an explicit alignment term, a rotation-invariant uniformity test gives a linear bound if and only if its spectrum dominates that of InfoNCE's kernel eβu⊤ v; SG's own test does, Gaussian kernels e−γ|u−v|2 qualify exactly when γ≥ β/2, and moment matching never does. Away from the optimum, the objectives differ. Along an isotropic nuisance channel, pure SG lowers its loss by adding per-view nuisance whenever the shared code is non-uniform. An alignment weight above the channel's gain makes the nuisance-free solution a strict local minimizer; for LeJEPA, the same rule gives a critical SIGReg weight that decreases with the batch size. At finite batch size, an off-diagonal U-statistic removes a plug-in bias toward misalignment. In controlled latent-variable models, pure SG retains per-view style, an alignment weight above the measured gain removes it, and for LeJEPA at three batch sizes the measured gain separates the encoders that retain style from those that do not. InfoNCE training also reaches a lower SG0.2 loss than SG0.2 training from scratch, which points to an optimization gap.
Related
- Anchor Divergence for Semantic Geometry in Contrastive Learning
- Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data
- Fisher Contrastive Learning to Mitigate Feature Suppression: A Case Study in Acute Promyelocytic Leukemia Diagnosis
- Joint Contrastive-Adversarial Representation Learning for Unsupervised Domain Adaptation
- Optimal VC Dimension of Contrastive Learning with Margin
BibTeX
@misc{zhao2026shared,
title = {{Shared Gaussianization: What Gaussian Regularizers Certify About Contrastive Learning, and What They Miss}},
author = {Ruoyu Zhao and Yuting Chen and Jinheng Zhang and Zhehao Zou and Tong Che},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10299},
eprint = {2610.10299},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10299},
}
RIS
TY - UNPB
TI - Shared Gaussianization: What Gaussian Regularizers Certify About Contrastive Learning, and What They Miss
AU - Ruoyu Zhao
AU - Yuting Chen
AU - Jinheng Zhang
AU - Zhehao Zou
AU - Tong Che
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10299
UR - https://arxiv.org/abs/2610.10299
AB - What can a distribution-matching regularizer such as SIGReg in LeJEPA certify about contrastive learning? We study shared Gaussianization (SG), a characteristic-function Gaussianity test on the average of two normalized views, scaled by an independent χ_d radius. Because disagreeing views shorten the average, one test detects both misalignment and non-uniformity. SG vanishes exactly at the aligned, uniform minimizers of population InfoNCE, and under equal marginals it bounds the InfoNCE excess by 4· 3^3/4β times the square root of the SG loss, plus a term linear in the loss. The square-root rate and this dimension-free constant are sharp, and no squared mean-embedding distance on view pairs achieves a faster rate. With an explicit alignment term, a rotation-invariant uniformity test gives a linear bound if and only if its spectrum dominates that of InfoNCE's kernel e^βu^⊤ v; SG's own test does, Gaussian kernels e^−γ|u−v|^2 qualify exactly when γ≥ β/2, and moment matching never does. Away from the optimum, the objectives differ. Along an isotropic nuisance channel, pure SG lowers its loss by adding per-view nuisance whenever the shared code is non-uniform. An alignment weight above the channel's gain makes the nuisance-free solution a strict local minimizer; for LeJEPA, the same rule gives a critical SIGReg weight that decreases with the batch size. At finite batch size, an off-diagonal U-statistic removes a plug-in bias toward misalignment. In controlled latent-variable models, pure SG retains per-view style, an alignment weight above the measured gain removes it, and for LeJEPA at three batch sizes the measured gain separates the encoders that retain style from those that do not. InfoNCE training also reaches a lower SG_0.2 loss than SG_0.2 training from scratch, which points to an optimization gap.
ER -