Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

Abstract

Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.

Effective Hamiltonian description on monitored Majorana chains: correlated power-law hoppings and unconventional entanglement scaling
Ken Mochizuki, Hisanori Oshima, Ryusuke Hamazaki +1
arXiv26 Oct 2026PreprintarXiv licenceFull textCondensed matter Quantum physics & quantum informationarXiv
On-shell renormalization in NREFT: a simplified description of resonances, bound states, and coupled channel effects
Guo-Ying Chen
arXiv30 Aug 2026PreprintarXiv licenceFull textHigh-energy & particle physics Nuclear physicsarXiv
Connecting finite-size scaling and renormalization-group flows in Anderson localization on random graphs
Ignacio García-Mata, Weitao Chen, John Martin +2
arXiv25 Sep 2026PreprintarXiv licenceFull textCondensed matter Quantum physics & quantum informationarXiv
Electroweak Thermal Histories in the Minimal Type-II Seesaw Model with a Renormalization Group-Improved Effective Potential
Cheng-Wei Chiang
arXiv1 Nov 2026PreprintCC BY 4.0Full textHigh-energy & particle physicsarXiv
The renormalization of the shell-model neutrinoless double-beta decay operator starting from effective field theory at leading order
L. Coraggio, G. De Gregorio, S. L. Lyu +1
arXiv1 Oct 2026PreprintCC BY 4.0Full textHigh-energy & particle physics Nuclear physicsarXiv

Cite

BibTeX RIS

BibTeX

@misc{chen2026scaling,
  title = {{Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking}},
  author = {Min Chen and Junqi Wang and Zeguan Wu and Wenhao Huang and Junyu Liu},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.36113},
  eprint = {2610.36113},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36113},
}

RIS

TY  - UNPB
TI  - Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking
AU  - Min Chen
AU  - Junqi Wang
AU  - Zeguan Wu
AU  - Wenhao Huang
AU  - Junyu Liu
PY  - 2026
DA  - 2026/10/26
PB  - arXiv
DO  - 10.48550/arxiv.2610.36113
UR  - https://arxiv.org/abs/2610.36113
AB  - Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.
ER  -

Abstract

Related

Cite