Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix
Abstract
The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Prior work argues that the top subspace of the Hessian stabilizes by measuring the overlap between the top subspaces of the step-wise Hessian, and explains this stabilization with diminishing parameter change in the late phase of training. In this paper, we define a new instability metric for the subspace evolution, and use it to detect subspace stabilization that is independent of the magnitude of parameter change. In the meantime, we observe that the gradient covariance matrix has a similar property of its top subspace to the Hessian. By using a between-class and within-class decomposition of the gradient covariance matrix, we identify an explicit form that gives a near-perfect approximation of the top-(C−1) subspace of the Hessian and the gradient covariance matrix. In the gradient flow set-up, we show that the slow evolution of the idenfied approximation is due to the separation between the outlier and the bulk eigenvalues of the Hessian matrix, thus providing an explanation to the phenomenon of the top subspace stabilization of the Hessian matrix.
Related
- Alternative proof of the gradient estimate for complex Hessian equations
- Gradient Estimates for Hessian Equations with Measure Data
- Gradient potential estimates for k-Hessian equations
- Existence and asymptotic behavior of radial solutions for a k-Hessian equation with the gradient
- Finite-Sample Approximation of Hessian-Guided Perturbed Wasserstein Gradient Flows
BibTeX
@misc{liao2026understanding,
title = {{Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix}},
author = {Fangshuo Liao and Anastasios Kyrillidis},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.31983},
eprint = {2609.31983},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.31983},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix
AU - Fangshuo Liao
AU - Anastasios Kyrillidis
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.31983
UR - https://arxiv.org/abs/2609.31983
AB - The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Prior work argues that the top subspace of the Hessian stabilizes by measuring the overlap between the top subspaces of the step-wise Hessian, and explains this stabilization with diminishing parameter change in the late phase of training. In this paper, we define a new instability metric for the subspace evolution, and use it to detect subspace stabilization that is independent of the magnitude of parameter change. In the meantime, we observe that the gradient covariance matrix has a similar property of its top subspace to the Hessian. By using a between-class and within-class decomposition of the gradient covariance matrix, we identify an explicit form that gives a near-perfect approximation of the top-(C−1) subspace of the Hessian and the gradient covariance matrix. In the gradient flow set-up, we show that the slow evolution of the idenfied approximation is due to the separation between the outlier and the bulk eigenvalues of the Hessian matrix, thus providing an explanation to the phenomenon of the top subspace stabilization of the Hessian matrix.
ER -