Fast and Stable Gradient Approximation for Quadratic Forms of Hermitian Matrix Functions
Abstract
Objectives involving quadratic forms u*f(A(θ))u for Hermitian A arise widely in scientific computing and machine learning. Lanczos efficiently approximates these quantities, but differentiating through its recurrence can introduce numerical instability and costly reorthogonalization. Building on existing low-rank Krylov approximations of Fréchet derivatives, we develop a matrix-free gradient implementation that reuses the forward Lanczos quantities without differentiating through the recurrence. We analyze its discrepancy from the exact gradient of the finite Lanczos approximation, deriving an exact residual-coupled error identity and a bound involving the best degree-(m−1) uniform polynomial approximation error of f', multiplied by a basis-sensitivity factor. The discrepancy vanishes when the Krylov subspace is invariant. Experiments demonstrate accurate gradients and runtime improvements over differentiated-Lanczos and modified-Arnoldi baselines. It remains numerically stable without reorthogonalization in all tested settings, including cases where adjoint-based Lanczos differentiation becomes unstable.
Related
- Stable Rational Approximation of PDE Transfer Functions with H∞ Error Bounds
- Hermitian K-theory for stable ∞-categories III: Grothendieck--Witt groups of rings
- A Galois-type Springer theorem for hermitian and generalized quadratic forms
- Stable moduli spaces of Hermitian forms
- Stability and gradient estimates for fully nonlinear elliptic equations on hermitian manifolds
BibTeX
@misc{singh2026fast,
title = {{Fast and Stable Gradient Approximation for Quadratic Forms of Hermitian Matrix Functions}},
author = {Navjot Singh and Kipton Barros and Xiaoye Sherry Li},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2605.12801},
eprint = {2605.12801},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.12801},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Fast and Stable Gradient Approximation for Quadratic Forms of Hermitian Matrix Functions
AU - Navjot Singh
AU - Kipton Barros
AU - Xiaoye Sherry Li
PY - 2026
DA - 2026/11/07
PB - arXiv
DO - 10.48550/arxiv.2605.12801
UR - https://arxiv.org/abs/2605.12801
AB - Objectives involving quadratic forms u^*f(A(θ))u for Hermitian A arise widely in scientific computing and machine learning. Lanczos efficiently approximates these quantities, but differentiating through its recurrence can introduce numerical instability and costly reorthogonalization. Building on existing low-rank Krylov approximations of Fréchet derivatives, we develop a matrix-free gradient implementation that reuses the forward Lanczos quantities without differentiating through the recurrence. We analyze its discrepancy from the exact gradient of the finite Lanczos approximation, deriving an exact residual-coupled error identity and a bound involving the best degree-(m−1) uniform polynomial approximation error of f', multiplied by a basis-sensitivity factor. The discrepancy vanishes when the Krylov subspace is invariant. Experiments demonstrate accurate gradients and runtime improvements over differentiated-Lanczos and modified-Arnoldi baselines. It remains numerically stable without reorthogonalization in all tested settings, including cases where adjoint-based Lanczos differentiation becomes unstable.
ER -