Derivative Computation in PINNs: Automatic Differentiation, Finite Differences and Beyond

Abstract

We systematically investigate finite-difference (FD) derivative computation in Physics-Informed Neural Networks (PINNs) as an alternative to automatic differentiation (AD). On three benchmark PDEs we show that, with a properly calibrated step size, FD matches AD in accuracy on every problem while running faster across the full tested batch-size range and using substantially less GPU memory, and that a stochastic variant we propose outperforms AD on a stationary problem. We further show that for neural architectures with inter-sample dependencies (e.g. BatchNorm, self-attention) the standard PyTorch autograd idiom is silently incorrect; the correct per-sample alternative is computationally infeasible at PINN-relevant batch sizes, while FD provides a forward-only approximation that is empirically an order of magnitude closer to the true per-sample derivative.

Shape calculus and automatic differentiation for multi-phase level-set topology optimisation with unfitted finite elements
Zachary J. Wegert, Martin Berggren, Vivien J. Challis
arXiv20 Oct 2026PreprintCC BY 4.0Full textNumerical analysis & scientific computing Optimisation & controlarXiv
Coherent differentiation with a monoid of degrees: derivatives, Taylor expansion, finite differences...
Thomas Ehrhard, Aymeric Walch
HAL20 Oct 2026PreprintCC BY-NC-SA 4.0Full textAlgebra & number theoryHALarXiv
Weighted composition--differentiation operators on mixed-norm, derivative Hardy and F(p,q,s) spaces
Jihua Sun, Junming Liu
arXiv14 Oct 2026PreprintCC BY-SA 4.0Full textAnalysis & PDEsarXiv
Coordinate-Free Automatic Differentiation
Alex Guerra, Joshua Aduol, Yaqian Tang +3
arXiv8 Oct 2026PreprintCC BY 4.0Full textGeometry & topologyarXiv
Automatic Differentiation with Constraints in Forward and Inverse Problems
Nikolay Yavich, Alexander Ryabov, Viacheslav Naumov +4
arXiv21 Oct 2026PreprintCC BY-NC-ND 4.0Full textNumerical analysis & scientific computingarXiv

Cite

BibTeX RIS

BibTeX

@misc{mikulski2026derivative,
  title = {{Derivative Computation in PINNs: Automatic Differentiation, Finite Differences and Beyond}},
  author = {Maciej J. Mikulski and Tadeusz Uhl},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2608.11020},
  eprint = {2608.11020},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.11020},
}

RIS

TY  - UNPB
TI  - Derivative Computation in PINNs: Automatic Differentiation, Finite Differences and Beyond
AU  - Maciej J. Mikulski
AU  - Tadeusz Uhl
PY  - 2026
DA  - 2026/10/31
PB  - arXiv
DO  - 10.48550/arxiv.2608.11020
UR  - https://arxiv.org/abs/2608.11020
AB  - We systematically investigate finite-difference (FD) derivative computation in Physics-Informed Neural Networks (PINNs) as an alternative to automatic differentiation (AD). On three benchmark PDEs we show that, with a properly calibrated step size, FD matches AD in accuracy on every problem while running faster across the full tested batch-size range and using substantially less GPU memory, and that a stochastic variant we propose outperforms AD on a stationary problem. We further show that for neural architectures with inter-sample dependencies (e.g. BatchNorm, self-attention) the standard PyTorch autograd idiom is silently incorrect; the correct per-sample alternative is computationally infeasible at PINN-relevant batch sizes, while FD provides a forward-only approximation that is empirically an order of magnitude closer to the true per-sample derivative.
ER  -

Abstract

Related

Cite