The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning
Abstract
In training a neural network with gradient descent (GD), each iteration induces a linear operator that governs the first-order updates to a model's internal state variables. We call this operator the Global Empirical Neural Tangent Kernel (GeNTK). In finite-width networks, GeNTK is typically intractable to form, leading prior work to focus on restrictive settings, such as tracking only outputs or infinite-width limits. Here, we bridge this gap by studying the structure of GeNTK across many models. Formulating the model state as the solution to a global implicit constraint, we derive GeNTK as a product of two operators: K, accounting for immediate parameter-to-state interactions, and P, describing internal state-to-state dependencies. For a broad class of "weight-based models", such as RNNs and transformers, we prove a universal Kronecker-core theorem showing that K admits an exact, forward-pass-computable form given by the Gram matrix of weight-site variables. This core gives a measurable state-dependent restriction on Euclidean GD, termed "self-referential bias": current activity filters which adjoint directions can drive a first-order update. For recurrent models (GRUs and RNNs), we examine the spectrum of GeNTK and formalize notions of low-rank, biased learning in terms of reduced spatial or temporal ranks of the full operator. Practically, this explains why some RNN initializations yield better learning, while some plateau, by biasing the GeNTK and thus restricting learning to a small domain of learnable tasks. To demonstrate computational viability, we validate the main theorems in a self-attention transformer and a pretrained Pythia transformer language model with 410 million parameters. Overall, our results expose tractable local rank and mode-selective biases in the finite-width GD geometry and open the door to principled study of the GeNTK, a bridge between theory and practice.
Related
- Learning controls in mean field games: the fictitious play and related gradient descent
- Continuous Delayed-Memory Stochastic Gradient Descent and Continuous-Time Reinforcement Learning from History of Astrophysical Time Series Studies
- Convergence of stochastic gradient descent for learning linear neural networks
- Comparative Performance Analysis of Optimization Solvers for Linear Support Vector Machines: Impact of Preconditioning and Dimensionality An Empirical Study Using the Benchopt Framework
- Broximal Gradient Descent: A Projection-Free Sister of Projected Gradient Descent
BibTeX
@misc{hazelden2026global,
title = {{The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning}},
author = {James Hazelden and Laura Driscoll and Eli Shlizerman and Eric Shea-Brown},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.08746},
eprint = {2605.08746},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.08746},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning
AU - James Hazelden
AU - Laura Driscoll
AU - Eli Shlizerman
AU - Eric Shea-Brown
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2605.08746
UR - https://arxiv.org/abs/2605.08746
AB - In training a neural network with gradient descent (GD), each iteration induces a linear operator that governs the first-order updates to a model's internal state variables. We call this operator the Global Empirical Neural Tangent Kernel (GeNTK). In finite-width networks, GeNTK is typically intractable to form, leading prior work to focus on restrictive settings, such as tracking only outputs or infinite-width limits. Here, we bridge this gap by studying the structure of GeNTK across many models. Formulating the model state as the solution to a global implicit constraint, we derive GeNTK as a product of two operators: K, accounting for immediate parameter-to-state interactions, and P, describing internal state-to-state dependencies. For a broad class of "weight-based models", such as RNNs and transformers, we prove a universal Kronecker-core theorem showing that K admits an exact, forward-pass-computable form given by the Gram matrix of weight-site variables. This core gives a measurable state-dependent restriction on Euclidean GD, termed "self-referential bias": current activity filters which adjoint directions can drive a first-order update. For recurrent models (GRUs and RNNs), we examine the spectrum of GeNTK and formalize notions of low-rank, biased learning in terms of reduced spatial or temporal ranks of the full operator. Practically, this explains why some RNN initializations yield better learning, while some plateau, by biasing the GeNTK and thus restricting learning to a small domain of learnable tasks. To demonstrate computational viability, we validate the main theorems in a self-attention transformer and a pretrained Pythia transformer language model with 410 million parameters. Overall, our results expose tractable local rank and mode-selective biases in the finite-width GD geometry and open the door to principled study of the GeNTK, a bridge between theory and practice.
ER -