Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry
Abstract
We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which nested architectural symmetries force subnetworks to merge in discrete blocks rather than by single-edge attachment. These structural transitions register as variance spikes in a macroscopic order parameter echoing physical phase transitions. We further state sufficient conditions under which the trapping argument carries over to Adam and AdamW under heavy-tailed gradient noise and measure them on a trained Transformer.
Related
- Organizing transitions and their cascades: Generalized symmetry enforcement in massless flows or Higgs transitions
- Modelling the complex dynamics of online disinformation cascades
- Transfer Dynamics and Spectral Cascades in Graph-Coupled Kuramoto Networks
- A Symplectic Theory of Turbulence Closure: Hidden Reservoir Dynamics, Endogenous Stochastic Transport, and Kraichnan Dual Cascades
- The Forgotten History of Wave Function Variance Optimization and its Relevance for Neural-Network VMC
BibTeX
@misc{ramachandran2026percolation,
title = {{Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry}},
author = {Sai Niranjan Ramachandran and Suvrit Sra},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.02373},
eprint = {2609.02373},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.02373},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry
AU - Sai Niranjan Ramachandran
AU - Suvrit Sra
PY - 2026
DA - 2026/09/27
PB - arXiv
DO - 10.48550/arxiv.2609.02373
UR - https://arxiv.org/abs/2609.02373
AB - We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which nested architectural symmetries force subnetworks to merge in discrete blocks rather than by single-edge attachment. These structural transitions register as variance spikes in a macroscopic order parameter echoing physical phase transitions. We further state sufficient conditions under which the trapping argument carries over to Adam and AdamW under heavy-tailed gradient noise and measure them on a trained Transformer.
ER -