Skip to content
DraconiticResearch
Search Sources API
194,161 studies31 sourcesUpdated 28 min ago
PreprintarXiv licencearXiv

Subspace Levenberg Marquardt Algorithms in Training Neural Networks

  • M. Duc Hoang
Open at sourcePDFDOICite

Abstract

The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-sized neural networks (NNs). However, its computational and memory costs increase significantly as the number of parameters in an NN grows. To address this limitation, subspace methods have been proposed, such as the Krylov subspace LM (KSLM) and the hybrid subspace LM (HSLM), making second-order algorithms more efficient. In this work, we evaluate the subspace Levenberg-Marquardt algorithms for regression and classification tasks in neural networks. We compare the performance of subspace LM variants with the classical LM method, as well as other popular first-order algorithms, such as stochastic gradient descent (SGD) and Adam.

Related

  1. Convergence of Levenberg--Marquardt method in Banach spaces
    Akari Ishida, Sei Nagayasu
    arXivPreprintarXiv licenceFull textAnalysis & PDEsNumerical analysis & scientific computingarXiv
  2. Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates
    Jianing Liu, Dong H. Zhang
    arXivPreprintarXiv licenceFull textPhysical chemistryNumerical analysis & scientific computingarXiv
  3. Hybrid Joint-Selective Optimization: Reduced-Space Levenberg-Marquardt Refinement of Low-Dimensional Parameters of Interest
    Muhammad Luthfi Shahab, Gabriella Alfa Indahsari, Imam Mukhlash +1
    arXivPreprintCC BY 4.0Full textNumerical analysis & scientific computingOptimisation & controlarXiv
  4. Symbolic Direct Construction of Augmented Systems for Levenberg-Marquardt Power Flow
    Tianshi Cheng, Venkata Dinavahi
    arXivPreprintarXiv licenceFull textNumerical analysis & scientific computingarXiv
  5. Training Variational Quantum Algorithms Is NP-Hard, Even Locally
    Dax Enshan Koh, Triscia Mundo, Iosif Sakos +1
    arXivPreprintarXiv licenceFull textOptimisation & controlQuantum physics & quantum informationarXiv

Cite

BibTeXRIS
BibTeX
@misc{hoang2026subspace,
  title = {{Subspace Levenberg Marquardt Algorithms in Training Neural Networks}},
  author = {M. Duc Hoang},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.00789},
  eprint = {2609.00789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.00789},
}
RIS
TY  - UNPB
TI  - Subspace Levenberg Marquardt Algorithms in Training Neural Networks
AU  - M. Duc Hoang
PY  - 2026
DA  - 2026/09/01
PB  - arXiv
DO  - 10.48550/arxiv.2609.00789
UR  - https://arxiv.org/abs/2609.00789
AB  - The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-sized neural networks (NNs). However, its computational and memory costs increase significantly as the number of parameters in an NN grows. To address this limitation, subspace methods have been proposed, such as the Krylov subspace LM (KSLM) and the hybrid subspace LM (HSLM), making second-order algorithms more efficient. In this work, we evaluate the subspace Levenberg-Marquardt algorithms for regression and classification tasks in neural networks. We compare the performance of subspace LM variants with the classical LM method, as well as other popular first-order algorithms, such as stochastic gradient descent (SGD) and Adam.
ER  -