An Analysis of Self-supervised Pre-training with Dependent Samples
Abstract
Self-supervised learning relies on so-called data augmentations ϕ(x) of unlabeled datapoints x --- for example, masking random pixels in an image x --- that should leave the label of x invariant and are often used to learn a lower-complexity invariant subspace cal V for downstream tasks. In practice, such augmentations { ϕl(xi) } are pooled together to learn cal V, despite obvious inter-dependencies between different augmentations ϕl(x), ϕk(x) of the same datapoint x. However, theoretical works on the subject typically consider procedures that avoid such dependencies, and are therefore limited to operate on smaller subsets of independent data. We show in this work that pooling augmentations together, despite inter-dependencies, is a better alternative than the baseline of partitioning the data into subsets of independent data. More precisely, in the context of estimating cal V, the statistical estimation error bounds for pooling are never worse than the partitioning baseline, and in some cases --- such as masking or noise injection-based augmentations over a shallow neural network --- naive pooling leads to faster rates in terms of the number of augmentations. The benefits of pooling are particularly prominent when the correlations between different augmentations ϕl(x), ϕk(x) have mild effects on estimation or help decrease the estimation variance. The analysis, therefore, yields new insights into the success of pooling augmented samples in self-supervised pre-training, and provides an intuition behind the practical preference towards using many augmentations.
Related
- From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Model
- How Many Independent Samples Does a Satellite Image Contain? Generalization Bounds for Spatially Dependent Data
- Survival Reinforcement Learning: Toward Scalable Self-Supervised RL
- Learning a Flow to Self-Supervised Representations
- Is Spectral Self-Supervised Learning a Minimax Optimal Method of Density Ratio Estimation?
BibTeX
@misc{fleissner2026analysis,
title = {{An Analysis of Self-supervised Pre-training with Dependent Samples}},
author = {Maximilian Fleissner and Debarghya Ghoshdastidar and Samory Kpotufe},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.05031},
eprint = {2609.05031},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.05031},
}
RIS
TY - UNPB
TI - An Analysis of Self-supervised Pre-training with Dependent Samples
AU - Maximilian Fleissner
AU - Debarghya Ghoshdastidar
AU - Samory Kpotufe
PY - 2026
DA - 2026/09/04
PB - arXiv
DO - 10.48550/arxiv.2609.05031
UR - https://arxiv.org/abs/2609.05031
AB - Self-supervised learning relies on so-called data augmentations ϕ(x) of unlabeled datapoints x --- for example, masking random pixels in an image x --- that should leave the label of x invariant and are often used to learn a lower-complexity invariant subspace cal V for downstream tasks. In practice, such augmentations { ϕ_l(x_i) } are pooled together to learn cal V, despite obvious inter-dependencies between different augmentations ϕ_l(x), ϕ_k(x) of the same datapoint x. However, theoretical works on the subject typically consider procedures that avoid such dependencies, and are therefore limited to operate on smaller subsets of independent data. We show in this work that pooling augmentations together, despite inter-dependencies, is a better alternative than the baseline of partitioning the data into subsets of independent data. More precisely, in the context of estimating cal V, the statistical estimation error bounds for pooling are never worse than the partitioning baseline, and in some cases --- such as masking or noise injection-based augmentations over a shallow neural network --- naive pooling leads to faster rates in terms of the number of augmentations. The benefits of pooling are particularly prominent when the correlations between different augmentations ϕ_l(x), ϕ_k(x) have mild effects on estimation or help decrease the estimation variance. The analysis, therefore, yields new insights into the success of pooling augmented samples in self-supervised pre-training, and provides an intuition behind the practical preference towards using many augmentations.
ER -