Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD
Abstract
Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining. We show that it can mispredict finite-batch stochastic gradient descent (SGD) qualitatively, and we trace the discrepancy to a specific mechanism. In a two-unit ReLU regression, a source task drives the two neurons toward positive proportionality and a target task rewards separating them. After source training for time T, gradient flow recovers on the target in time linear in T. Online SGD with batch size b and step size η in both phases instead fails with high probability throughout a horizon of order ec/η once T ≳ log(b/η), uniformly on an explicit set of initializations with Gaussian probability above one percent. For each fixed T, small-step SGD still recovers, so the failure requires the joint limit of small steps and long pretraining. At the target clone, the population instability is carried entirely by inputs on which the two ReLU gates disagree. For units at angle δ these inputs form a wedge of probability δ/π, and weight decay shrinks the angle exponentially during pretraining. On every other input both units receive the same random linear update, which contracts their separation in conditional expectation. Bounding the cumulative probability of sampling the wedge along the exact online recursion, without a diffusion approximation, shows that recovery with fixed probability from an identical source-gradient-flow checkpoint, within ec/η updates, requires Nb ≳ eλT target samples and batch size b ≳ ηeλT, where N counts updates and λ is the weight decay. In simulations, recovery is approximately a function of the disagreement budget bδ/η and saturates in the horizon.
Related
- A functional central limit theorem for kernel gradient flow and infinitesimal gradient boosting
- Mean-Field Oscillator Ising Machines: Gradient Flows and Classification of Limit Solutions
- A q-recurrence for a finite Apéry limit
- Spectral embedding through weak* limit of finite-dimensional approximations
- An interior penalty nonconforming finite element method for strain gradient elasticity
BibTeX
@misc{zhao2026rare,
title = {{Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD}},
author = {Ruoyu Zhao and Mingxuan Zhang and Jianbo Dai and Jiaqi Wu and Chenyu Zhu and Tong Che},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.11475},
eprint = {2610.11475},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.11475},
}
RIS
TY - UNPB
TI - Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD
AU - Ruoyu Zhao
AU - Mingxuan Zhang
AU - Jianbo Dai
AU - Jiaqi Wu
AU - Chenyu Zhu
AU - Tong Che
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.11475
UR - https://arxiv.org/abs/2610.11475
AB - Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining. We show that it can mispredict finite-batch stochastic gradient descent (SGD) qualitatively, and we trace the discrepancy to a specific mechanism. In a two-unit ReLU regression, a source task drives the two neurons toward positive proportionality and a target task rewards separating them. After source training for time T, gradient flow recovers on the target in time linear in T. Online SGD with batch size b and step size η in both phases instead fails with high probability throughout a horizon of order e^c/η once T ≳ log(b/η), uniformly on an explicit set of initializations with Gaussian probability above one percent. For each fixed T, small-step SGD still recovers, so the failure requires the joint limit of small steps and long pretraining. At the target clone, the population instability is carried entirely by inputs on which the two ReLU gates disagree. For units at angle δ these inputs form a wedge of probability δ/π, and weight decay shrinks the angle exponentially during pretraining. On every other input both units receive the same random linear update, which contracts their separation in conditional expectation. Bounding the cumulative probability of sampling the wedge along the exact online recursion, without a diffusion approximation, shows that recovery with fixed probability from an identical source-gradient-flow checkpoint, within e^c/η updates, requires Nb ≳ e^λT target samples and batch size b ≳ ηe^λT, where N counts updates and λ is the weight decay. In simulations, recovery is approximately a function of the disagreement budget bδ/η and saturates in the horizon.
ER -