ReLU-spline activation functions for Neural Network approximation of solutions to differential equations
Abstract
We introduce the family of ReLU-Spline activation of arbitrary order. These functions are piecewise polynomial and globally Lipschitz. They inherit the approximation property of polynomial splines S2r,r. Then we show that they can be used to approximation the solution of various differential elliptic PDEs with DeepRitz, PINNs and VPINNs methods. Since the training brings important constraints in terms of the regularity of activation functions, we formulate the principle that the minimal regularity of an activation function must be equal to the number of derivatives in the formulation plus one for the training. As a consequence of this principle, the ReLU activation function cannot approximate the solution of elliptic PDEs at the end of a generic training sequence. A solution is to use higher order ReLU-spline activation functions. We provide numerical illustrations in 1D and 2D which validate the theory.
Related
- Global solutions to the Boussinesq equations and the planetary geostrophic approximation
- Approximation of Random Differential Equations Driven by Physical Brownian Motion with Fast Oscillating Noise
- Periodic measures for time-inhomogeneous stochastic differential equations driven by multiplicative α-stable processes: Existence, uniqueness and numerical approximation
- Meromorphic Solutions of Certain Kind of Non-Linear Differential equation with finite sums of Exponential Functions
- Existence and Nonexistence of Solutions of Certain type of Nonlinear Differential and Differential-Difference Equations
BibTeX
@misc{despres2026reluspline,
title = {{ReLU-spline activation functions for Neural Network approximation of solutions to differential equations}},
author = {Bruno Després and Moreno Pintore},
howpublished = {HAL},
year = {2026},
month = oct,
url = {https://hal.sorbonne-universite.fr/hal-05809525v1},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - ReLU-spline activation functions for Neural Network approximation of solutions to differential equations
AU - Bruno Després
AU - Moreno Pintore
PY - 2026
DA - 2026/10/19
PB - HAL
UR - https://hal.sorbonne-universite.fr/hal-05809525v1
AB - We introduce the family of ReLU-Spline activation of arbitrary order. These functions are piecewise polynomial and globally Lipschitz. They inherit the approximation property of polynomial splines S2r,r. Then we show that they can be used to approximation the solution of various differential elliptic PDEs with DeepRitz, PINNs and VPINNs methods. Since the training brings important constraints in terms of the regularity of activation functions, we formulate the principle that the minimal regularity of an activation function must be equal to the number of derivatives in the formulation plus one for the training. As a consequence of this principle, the ReLU activation function cannot approximate the solution of elliptic PDEs at the end of a generic training sequence. A solution is to use higher order ReLU-spline activation functions. We provide numerical illustrations in 1D and 2D which validate the theory.
ER -