Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

Abstract

Overparameterized networks often have ill-conditioned, saddle rich loss landscapes. They are usually trained with first-order methods that struggle on those landscapes, while many second-order methods stay infeasible at scale. Some scalable cubic methods replace the Hessian by a computationally cheap approximation and drop the exact block curvature. This paper introduces ARC-block, a blockwise adaptive regularization with cubics (ARC) optimizer. For each parameter tensor it minimizes a cubic model over the true block Hessian, with a per-block adaptive cubic constant and a monotone guard on the full loss. Large tensors are handled matrix-free in a Lanczos-built Krylov subspace. ARC-block's CubicKrylov subsolver's descent is monotone with a per-block cubic decrease bound. For exact steps, and for Krylov steps that meet the Cartis--Gould--Toint residual test, we provide an 𝒪(√Bε^−3/2) sweep bound above a cross-block coupling floor. Another subsolver, ARC-φ₁, uses a trust-controlled exponential relaxation step. Among the cubic Newton methods evaluated here, these are the only ones whose steps stay exact on every parameter block (within the Krylov subspace), including on a $91.4$M-parameter implicit neural representation (INR). Run to convergence on FINER 2D image-fitting INR network, ARC-φ₁ reaches $133.5$ dB peak signal-to-noise ratio (PSNR), with no post peak collapse. Tuned Adam peaks at $78.2$ dB at 2,629 s (step ∼625k, about $44$ minutes), further steps degrade its PSNR. On the same benchmark, SOAP, L-BFGS, and the scalable diagonal cubic method AdaCubic stay at or below Adam's $78.2$ dB peak. Over the benchmark's 20 images, run to a plateau rule, ARC-φ₁'s per-image PSNR has median $132.4$ and mean 131.5±3.3 dB against tuned Adam's 72.8±4.5 dB.

On the Accuracy of Approximate Leave-One-Out with Blockwise Nonsmooth Regularization in High-dimensions
Honghao Lin, Songbin Liu, Arian Maleki +5
arXiv20 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Phase Transition in Binary Compressed Sensing via Annealing with Adaptive Regularization
Xiaoxin Huang, Masayuki Ohzeki
arXiv15 Sep 2026PreprintarXiv licenceFull textGeneral mathematics Quantum physics & quantum informationarXiv
Sparse Signal Recovery Using Log-Sum Regularization and Adaptive Smoothing
Keisuke Morita, Masayuki Ohzeki
arXiv3 Sep 2026PreprintCC BY 4.0Full textGeneral mathematicsarXiv
Adaptive Regularization via Extreme Value Distributions for Gaussian Graphical Models
Alexander P. Christensen, Jeongwon Choi, Haoyi Yang +1
arXiv4 Sep 2026PreprintCC BY 4.0Full textStatisticsarXiv
DREAM: Deep-Reparametrization of Adaptive Regularization Maps for Fast Zero-Shot Self-Supervised Learning
Thanh Trung Vu, Ander Biguri, Christoph Kolbitsch +3
arXiv3 Sep 2026PreprintCC BY-NC-ND 4.0Full textOptimisation & controlarXiv

Cite

BibTeX RIS

BibTeX

@misc{podorozhny2026blockwise,
  title = {{Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence}},
  author = {Rodion Podorozhny},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2608.22129},
  eprint = {2608.22129},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.22129},
}

RIS

TY  - UNPB
TI  - Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence
AU  - Rodion Podorozhny
PY  - 2026
DA  - 2026/10/14
PB  - arXiv
DO  - 10.48550/arxiv.2608.22129
UR  - https://arxiv.org/abs/2608.22129
AB  - Overparameterized networks often have ill-conditioned, saddle rich loss landscapes. They are usually trained with first-order methods that struggle on those landscapes, while many second-order methods stay infeasible at scale. Some scalable cubic methods replace the Hessian by a computationally cheap approximation and drop the exact block curvature. This paper introduces ARC-block, a blockwise adaptive regularization with cubics (ARC) optimizer. For each parameter tensor it minimizes a cubic model over the true block Hessian, with a per-block adaptive cubic constant and a monotone guard on the full loss. Large tensors are handled matrix-free in a Lanczos-built Krylov subspace. ARC-block's CubicKrylov subsolver's descent is monotone with a per-block cubic decrease bound. For exact steps, and for Krylov steps that meet the Cartis--Gould--Toint residual test, we provide an 𝒪(√Bε^−3/2) sweep bound above a cross-block coupling floor. Another subsolver, ARC-φ_1, uses a trust-controlled exponential relaxation step. Among the cubic Newton methods evaluated here, these are the only ones whose steps stay exact on every parameter block (within the Krylov subspace), including on a $91.4$M-parameter implicit neural representation (INR). Run to convergence on FINER 2D image-fitting INR network, ARC-φ_1 reaches $133.5$ dB peak signal-to-noise ratio (PSNR), with no post peak collapse. Tuned Adam peaks at $78.2$ dB at 2,629 s (step ∼625k, about $44$ minutes), further steps degrade its PSNR. On the same benchmark, SOAP, L-BFGS, and the scalable diagonal cubic method AdaCubic stay at or below Adam's $78.2$ dB peak. Over the benchmark's 20 images, run to a plateau rule, ARC-φ_1's per-image PSNR has median $132.4$ and mean 131.5±3.3 dB against tuned Adam's 72.8±4.5 dB.
ER  -

Abstract

Related

Cite