PowerStep: Memory-Efficient Adaptive Optimization via ℓp-Norm Steepest Descent
Abstract
Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by ℓp-norm steepest descent, PowerStep applies a signed-power transform directly to one momentum buffer. We establish a finite-horizon stationarity bound for exact, unregularized updates, with an O(1/√T) term and a noise-dependent residual. Experiments on Transformers from 124M to 235B parameters show competitive validation quality while halving fp32 optimizer-state memory relative to AdamW. Combined with uniform int8 quantization, PowerStep remains numerically stable and reduces optimizer-state memory by ∼8× compared to fp32 AdamW. PowerStep thus provides a simple, memory-efficient alternative for large-scale training.
Related
- Counterexamples to R-superlinear convergence of cyclic steepest descent
- Convergence of cyclic steepest descent for quadratics with two distinct eigenvalues
- ABrA-GD: Adaptive Bregman Accelerated Gradient Descent for Relatively Smooth and (Strongly-)Convex Optimization
- Doubly exponential convergence of the cyclic steepest descent method for strictly convex quadratics in arbitrary dimensions
- Convex optimization on moment polytopes: Hadamard mirror descent and efficient algorithms for quantum functionals and other tensor parameters
BibTeX
@misc{lu2026powerstep,
title = {{PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$-Norm Steepest Descent}},
author = {Yao Lu and Dengdong Fan and Shixun Zhang and Yonghong Tian},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.10335},
eprint = {2605.10335},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.10335},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - PowerStep: Memory-Efficient Adaptive Optimization via ℓ_p-Norm Steepest Descent
AU - Yao Lu
AU - Dengdong Fan
AU - Shixun Zhang
AU - Yonghong Tian
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2605.10335
UR - https://arxiv.org/abs/2605.10335
AB - Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by ℓ_p-norm steepest descent, PowerStep applies a signed-power transform directly to one momentum buffer. We establish a finite-horizon stationarity bound for exact, unregularized updates, with an O(1/√T) term and a noise-dependent residual. Experiments on Transformers from 124M to 235B parameters show competitive validation quality while halving fp32 optimizer-state memory relative to AdamW. Combined with uniform int8 quantization, PowerStep remains numerically stable and reduces optimizer-state memory by ∼8× compared to fp32 AdamW. PowerStep thus provides a simple, memory-efficient alternative for large-scale training.
ER -