Mini-Batch Pruning Techniques in Stochastic Gradient Descent Methods
- 1Department of Mathematics “Giuseppe Peano”, University of Torino, Via Carlo Alberto 10, 10123 Torino, Italy
Abstract
Mini-batch Stochastic Gradient Descent (SGD) is a standard optimization method for training neural networks and other large-scale parametric models. We study three loss-based mini-batch pruning strategies—Perceptron-Inspired SGD (PISGD), Pruning SGD (PSGD), and No-Release PSGD (NRPSGD)—that reduce the number of per-sample gradient evaluations by deprioritizing low-loss samples during training. The methods are evaluated in Python on two problems: polynomial function approximation and MNIST handwritten-digit classification with a one-hidden-layer neural network. We examine the sensitivity to pruning hyperparameters and compare the three strategies with standard mini-batch SGD in terms of training time and optimization or predictive performance. In the reported MNIST experiments, the pruning methods show competitive or improved time–accuracy and time–loss behavior relative to standard mini-batch SGD, with NRPSGD obtaining the most favorable results in the longer runs. These findings provide proof-of-concept evidence restricted to the experimental settings considered. They indicate that loss-based mini-batch pruning can reduce gradient-computation cost without degrading solution quality in the tested settings. Because the pruning mechanism leaves the parameter-update rule unchanged, it may also be combined with adaptive stochastic optimizers such as RMSProp and Adam.
Related
- A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus
- Convergence rate of stochastic gradient descent under Łojasiewicz type condition and polynomially controlled noise
- Exact Theory and Optimal Stepsizes for Stochastic Gradient Descent in Smooth Convex Minimization
- Continuous Delayed-Memory Stochastic Gradient Descent and Continuous-Time Reinforcement Learning from History of Astrophysical Time Series Studies
- Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
BibTeX
@article{cavoretto2026minibatch,
title = {{Mini-Batch Pruning Techniques in Stochastic Gradient Descent Methods}},
author = {Roberto Cavoretto and Alessandra De Rossi and Enrico Scibilia},
journal = {Mathematics},
year = {2026},
month = oct,
doi = {10.3390/math14193565},
url = {https://www.mdpi.com/2227-7390/14/19/3565},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Mini-Batch Pruning Techniques in Stochastic Gradient Descent Methods
AU - Roberto Cavoretto
AU - Alessandra De Rossi
AU - Enrico Scibilia
PY - 2026
DA - 2026/10/01
JO - Mathematics
DO - 10.3390/math14193565
UR - https://www.mdpi.com/2227-7390/14/19/3565
AB - Mini-batch Stochastic Gradient Descent (SGD) is a standard optimization method for training neural networks and other large-scale parametric models. We study three loss-based mini-batch pruning strategies—Perceptron-Inspired SGD (PISGD), Pruning SGD (PSGD), and No-Release PSGD (NRPSGD)—that reduce the number of per-sample gradient evaluations by deprioritizing low-loss samples during training. The methods are evaluated in Python on two problems: polynomial function approximation and MNIST handwritten-digit classification with a one-hidden-layer neural network. We examine the sensitivity to pruning hyperparameters and compare the three strategies with standard mini-batch SGD in terms of training time and optimization or predictive performance. In the reported MNIST experiments, the pruning methods show competitive or improved time–accuracy and time–loss behavior relative to standard mini-batch SGD, with NRPSGD obtaining the most favorable results in the longer runs. These findings provide proof-of-concept evidence restricted to the experimental settings considered. They indicate that loss-based mini-batch pruning can reduce gradient-computation cost without degrading solution quality in the tested settings. Because the pruning mechanism leaves the parameter-update rule unchanged, it may also be combined with adaptive stochastic optimizers such as RMSProp and Adam.
SN - 2227-7390
ER -