GPU-Enabled Large-Scale Optimization Using Randomized Linear Algebra
Abstract
This paper introduces rlaopt, a PyTorch-based package for large-scale optimization and scientific computing using randomized numerical linear algebra (RandNLA). Despite substantial progress in RandNLA-based algorithms, few implementations combine GPU acceleration with a simple interface for specifying optimization problems. rlaopt addresses this gap by providing GPU-enabled solvers for positive-definite linear systems and convex empirical risk minimization with constraints and regularizers. These solvers use RandNLA to accelerate conjugate gradient (NystromPCG), operator splitting (NysADMM), and stochastic gradient methods (SAPPHIRE). Moreover, rlaopt includes a modeling language that lets users specify problems using natural mathematical syntax. rlaopt automatically checks compatibility with the selected solver and performs the required problem decomposition. The solvers also support differentiation through their iterations, enabling applications such as hyperparameter tuning. Experiments on ridge regression, bounded multinomial logistic regression, and bounded elastic net identify when randomized preconditioning improves performance and demonstrate substantial speedups from GPU execution. The package is open-source under an Apache license, with source code at https://github.com/udellgroup/rlaopt and version 0.1.0 available on PyPI.
Related
- Randomized algorithms in numerical linear algebra for scientific computing
- The Numerical Linear Algebra of Large Language Models
- Lower Bounds for Randomized Linear-Oracle Optimization over Strongly Convex Sets
- Extending structures for the linear framework of Universal Algebra with applications to transposed Poisson algebras
- Advanced Linear Algebra with Applications - Part I (Numerical linear algebra for PDEs, machine learning, and data assimilation)
BibTeX
@misc{rathore2026gpuenabled,
title = {{GPU-Enabled Large-Scale Optimization Using Randomized Linear Algebra}},
author = {Pratik Rathore and Zachary Frangella and Parth Nobel and Xuning Hu and Madeleine Udell},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.08136},
eprint = {2609.08136},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.08136},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - GPU-Enabled Large-Scale Optimization Using Randomized Linear Algebra
AU - Pratik Rathore
AU - Zachary Frangella
AU - Parth Nobel
AU - Xuning Hu
AU - Madeleine Udell
PY - 2026
DA - 2026/09/08
PB - arXiv
DO - 10.48550/arxiv.2609.08136
UR - https://arxiv.org/abs/2609.08136
AB - This paper introduces rlaopt, a PyTorch-based package for large-scale optimization and scientific computing using randomized numerical linear algebra (RandNLA). Despite substantial progress in RandNLA-based algorithms, few implementations combine GPU acceleration with a simple interface for specifying optimization problems. rlaopt addresses this gap by providing GPU-enabled solvers for positive-definite linear systems and convex empirical risk minimization with constraints and regularizers. These solvers use RandNLA to accelerate conjugate gradient (NystromPCG), operator splitting (NysADMM), and stochastic gradient methods (SAPPHIRE). Moreover, rlaopt includes a modeling language that lets users specify problems using natural mathematical syntax. rlaopt automatically checks compatibility with the selected solver and performs the required problem decomposition. The solvers also support differentiation through their iterations, enabling applications such as hyperparameter tuning. Experiments on ridge regression, bounded multinomial logistic regression, and bounded elastic net identify when randomized preconditioning improves performance and demonstrate substantial speedups from GPU execution. The package is open-source under an Apache license, with source code at https://github.com/udellgroup/rlaopt and version 0.1.0 available on PyPI.
ER -