Skip to content
DraconiticResearch
Search Sources API
203,234 studies31 sourcesUpdated 33 min ago
Peer-reviewedCC BYCommunications on Pure and Applied Mathematics

Stochastic Gradient Descent in High Dimensions for Multi‐Spiked Tensor PCA

  • Gérard Ben Arous1,2
  • Cédric Gerbelot1,2,3
  • Vanessa Piccolo4,3
4 affiliations
  1. 1Courant Institute of Mathematical Sciences New York University New York City New York USA
  2. 2CIMS - Courant Institute of Mathematical Sciences [New York] (251, Mercier Street, New York, NY 10012 - United States)
  3. 3UMPA-ENSL - Unité de Mathématiques Pures et Appliquées (France)
  4. 4Unité de Mathématiques Pures et Appliquées (UMPA) ENS Lyon Lyon France
Open at sourceDOICite

Abstract

We study the high‐dimensional dynamics of online stochastic gradient descent (SGD) for the multi‐spiked tensor model. This multi‐index model arises from the tensor principal component analysis (PCA) problem with multiple spikes, where the goal is to estimate the unknown signal vectors within the ‐dimensional unit sphere through maximum likelihood estimation from noisy observations of a ‐tensor. We determine the number of samples and the conditions on the signal‐to‐noise ratios (SNRs) required to efficiently recover the unknown spikes from natural random initializations. We show that full recovery of all spikes is possible provided a number of sample scaling as , matching the algorithmic threshold identified in the rank‐one case. Our results are obtained through a detailed analysis of a low‐dimensional system that describes the evolution of the correlations between the estimators and the spikes, while sharply controlling the noise in the dynamics. We find the spikes are recovered sequentially in a process we term “sequential elimination”: once a correlation exceeds a critical threshold, all correlations sharing a row or column index become sufficiently small, allowing the next correlation to grow and become macroscopic. The order in which correlations become macroscopic depends on their initial values and the corresponding SNRs, leading to either exact recovery or recovery of a permutation of the spikes. In the matrix case, when , if the SNRs are sufficiently separated, we achieve exact recovery of the spikes, whereas equal SNRs lead to recovery of the subspace spanned by them.

Related

  1. A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus
    Neil Walton
    arXivPreprintCC BY 4.0Full textOptimisation & controlProbabilityarXiv
  2. Convergence rate of stochastic gradient descent under Łojasiewicz type condition and polynomially controlled noise
    Paweł Węgrzyn
    arXivPreprintarXiv licenceFull textNumerical analysis & scientific computingOptimisation & controlarXiv
  3. Exact Theory and Optimal Stepsizes for Stochastic Gradient Descent in Smooth Convex Minimization
    Benjamin Grimmer, Qining Shen
    arXivPreprintarXiv licenceFull textOptimisation & controlarXiv
  4. Continuous Delayed-Memory Stochastic Gradient Descent and Continuous-Time Reinforcement Learning from History of Astrophysical Time Series Studies
    Juncheng Yi
    arXivPreprintCC BY 4.0Full textStatisticsarXiv
  5. Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
    Zijun Chen, Zihan Zhang
    arXivPreprintarXiv licenceFull textOptimisation & controlStatisticsarXiv

Cite

BibTeXRIS
BibTeX
@article{arous2026stochastic,
  title = {{Stochastic Gradient Descent in High Dimensions for Multi‐Spiked Tensor PCA}},
  author = {Gérard Ben Arous and Cédric Gerbelot and Vanessa Piccolo},
  journal = {Communications on Pure and Applied Mathematics},
  year = {2026},
  month = jun,
  doi = {10.1002/cpa.70056},
  url = {https://doi.org/10.1002/cpa.70056},
  note = {CC BY},
}
RIS
TY  - JOUR
TI  - Stochastic Gradient Descent in High Dimensions for Multi‐Spiked Tensor PCA
AU  - Gérard Ben Arous
AU  - Cédric Gerbelot
AU  - Vanessa Piccolo
PY  - 2026
DA  - 2026/06/24
JO  - Communications on Pure and Applied Mathematics
DO  - 10.1002/cpa.70056
UR  - https://doi.org/10.1002/cpa.70056
AB  - We study the high‐dimensional dynamics of online stochastic gradient descent (SGD) for the multi‐spiked tensor model. This multi‐index model arises from the tensor principal component analysis (PCA) problem with multiple spikes, where the goal is to estimate the unknown signal vectors within the ‐dimensional unit sphere through maximum likelihood estimation from noisy observations of a ‐tensor. We determine the number of samples and the conditions on the signal‐to‐noise ratios (SNRs) required to efficiently recover the unknown spikes from natural random initializations. We show that full recovery of all spikes is possible provided a number of sample scaling as , matching the algorithmic threshold identified in the rank‐one case. Our results are obtained through a detailed analysis of a low‐dimensional system that describes the evolution of the correlations between the estimators and the spikes, while sharply controlling the noise in the dynamics. We find the spikes are recovered sequentially in a process we term “sequential elimination”: once a correlation exceeds a critical threshold, all correlations sharing a row or column index become sufficiently small, allowing the next correlation to grow and become macroscopic. The order in which correlations become macroscopic depends on their initial values and the corresponding SNRs, leading to either exact recovery or recovery of a permutation of the spikes. In the matrix case, when , if the SNRs are sufficiently separated, we achieve exact recovery of the spikes, whereas equal SNRs lead to recovery of the subspace spanned by them.
SN  - 0010-3640
SN  - 1097-0312
ER  -