A sequential regularized piecewise affine algorithm for nonconvex nonsmooth multicomposite optimization in RNN training
Abstract
This paper focuses on a class of nonconvex nonsmooth multicomposite optimization problems for training RNNs (Recurrent Neural Networks). We first establish easily verifiable conditions under which an approximate first-order d-stationary point of the problem is guaranteed to be an approximate second-order d-stationary point. Subsequently, we propose a sequential regularized piecewise affine algorithm, SRPA, that minimizes a regularized piecewise affine function at each iteration, where the idea of active-set strategy is incorporated to reduce the computational cost per iteration. Leveraging the aforementioned conditions for second-order d-stationarity, we establish the global convergence of SRPA to second-order d-stationary points and the complexity bound of $ \mathcal{O}(ε^{-2}) $ for obtaining an $ ε$-approximate second-order d-stationary point. Finally, numerical experiments for training RNNs on synthetic and real-world datasets demonstrate the promising performance of SRPA compared with state-of-the-art algorithms.
Related
- Normal form method of center-focus problem in piecewise-smooth systems and algorithm design
- Accelerated Prox-Level Methods for Unknown Piecewise-Smooth Optimization II: Function-constrained Optimization
- Regularized Sequential Reconstructions of Dynamic Nonsmooth Sources from Sparse Boundary Flux Data
- Stabilized SQP Method for Convex Piecewise Linear-Quadratic Composite Optimization Problems
- B3-PWL: GPU-Batched Branch-and-Bound for Piecewise-Linear Optimization with SOS2 Constraints
BibTeX
@misc{jin2026sequential,
title = {{A sequential regularized piecewise affine algorithm for nonconvex nonsmooth multicomposite optimization in RNN training}},
author = {Lingzi Jin and Xiao Wang and Xiaojun Chen},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.18325},
eprint = {2609.18325},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.18325},
}
RIS
TY - UNPB
TI - A sequential regularized piecewise affine algorithm for nonconvex nonsmooth multicomposite optimization in RNN training
AU - Lingzi Jin
AU - Xiao Wang
AU - Xiaojun Chen
PY - 2026
DA - 2026/09/16
PB - arXiv
DO - 10.48550/arxiv.2609.18325
UR - https://arxiv.org/abs/2609.18325
AB - This paper focuses on a class of nonconvex nonsmooth multicomposite optimization problems for training RNNs (Recurrent Neural Networks). We first establish easily verifiable conditions under which an approximate first-order d-stationary point of the problem is guaranteed to be an approximate second-order d-stationary point. Subsequently, we propose a sequential regularized piecewise affine algorithm, SRPA, that minimizes a regularized piecewise affine function at each iteration, where the idea of active-set strategy is incorporated to reduce the computational cost per iteration. Leveraging the aforementioned conditions for second-order d-stationarity, we establish the global convergence of SRPA to second-order d-stationary points and the complexity bound of $ \mathcal{O}(ε^{-2}) $ for obtaining an $ ε$-approximate second-order d-stationary point. Finally, numerical experiments for training RNNs on synthetic and real-world datasets demonstrate the promising performance of SRPA compared with state-of-the-art algorithms.
ER -