Accelerating natural gradient descent for PINNs with randomized numerical linear algebra
Abstract
Natural Gradient Descent (NGD) has emerged as a promising optimization algorithm for training neural network-based solvers for partial differential equations (PDEs), such as Physics-Informed Neural Networks (PINNs). However, its practical use is often limited by the high computational cost of solving linear systems involving the Gramian matrix. While matrix-free NGD methods based on the conjugate gradient (CG) method avoid explicit matrix inversion, the ill-conditioning of the Gramian significantly slows the convergence of CG. In this work, we extend matrix-free NGD to broader classes of problems than previously considered and propose the use of Randomized Numerical Linear Algebra (RandNLA) techniques for efficient preconditioning of the inner CG solver. The resulting algorithms demonstrate substantial performance improvements over existing NGD-based methods on a range of PDE problems discretized using neural networks, and offer competitive results compared to other state-of-the-art optimizers.
Related
- Camera-in-the-loop gradient descent hologram generation for Fraunhofer diffraction
- The Bondi dipole in full numerical relativity: a self-accelerating positive-negative mass binary
- Stochastic learning flow: moment equations for stochastic gradient descent at finite step size
- Rank-one Factored Gradient Descent without Restricted Isometry: Application to Quantum Pure State Tomography on Multi-Qubits
- Collective-Variable Stein Variational Gradient Descent for Free Energy Surface Exploration
BibTeX
@article{bioli2026accelerating,
title = {{Accelerating natural gradient descent for PINNs with randomized numerical linear algebra}},
author = {Ivan Bioli and Carlo Marcati and Giancarlo Sangalli},
journal = {Computer Methods in Applied Mechanics and Engineering},
year = {2026},
month = oct,
doi = {10.1016/j.cma.2026.119456},
url = {https://hal.science/hal-05840428v1},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Accelerating natural gradient descent for PINNs with randomized numerical linear algebra
AU - Ivan Bioli
AU - Carlo Marcati
AU - Giancarlo Sangalli
PY - 2026
DA - 2026/10/30
JO - Computer Methods in Applied Mechanics and Engineering
DO - 10.1016/j.cma.2026.119456
UR - https://hal.science/hal-05840428v1
AB - Natural Gradient Descent (NGD) has emerged as a promising optimization algorithm for training neural network-based solvers for partial differential equations (PDEs), such as Physics-Informed Neural Networks (PINNs). However, its practical use is often limited by the high computational cost of solving linear systems involving the Gramian matrix. While matrix-free NGD methods based on the conjugate gradient (CG) method avoid explicit matrix inversion, the ill-conditioning of the Gramian significantly slows the convergence of CG. In this work, we extend matrix-free NGD to broader classes of problems than previously considered and propose the use of Randomized Numerical Linear Algebra (RandNLA) techniques for efficient preconditioning of the inner CG solver. The resulting algorithms demonstrate substantial performance improvements over existing NGD-based methods on a range of PDE problems discretized using neural networks, and offer competitive results compared to other state-of-the-art optimizers.
SN - 0045-7825
SN - 1879-2138
ER -