Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
Abstract
Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients. The pursuit of gradient-free methods has long been a goal in the field of artificial intelligence. This paper shows that indeed the simplest Monte Carlo algorithm implemented on a single GPU -- randomly mutate a parameter, keep it if the loss decreases, otherwise retry -- can practically train deep networks. This gradient-free method does not even need common techniques such as batch normalization or residual connections to directly train sufficiently deep networks. More remarkably, its flexibility extends to several nontrivial scenarios: it enables pure pruning training, supports discrete weights, accommodates unconventional transfer functions such as Gaussian, and reveals the substantial redundancy of deep networks. We have demonstrated its feasibility on deep networks with more than 20 layers, single-hidden-layer wide networks with up to 16,384 hidden neurons, and even a simple Transformer architecture trained on both image classification (MNIST) and character-level language modeling (Tiny Shakespeare). This simple gradient-free method may offer a complementary perspective for understanding the self-organization and learning mechanisms of neural networks, and also provides an alternative route for building physically inspired deep learning systems.
Versions
Related
- A samplewise backpropagation method for neural networks driven by fractional Brownian motion
- Exactly explicit solutions of (2+1)-dimensional conformable fractional diffusive Predator–Prey model via neural networks method
- Two-level domain-decomposition AdaGrad method for scalable training of graph neural networks
- A Hybrid Physics-informed holomorphic neural networks Method
- Hybrid coupling with numerics-informed neural networks and the overlapping Schwarz alternating method
BibTeX
@misc{zhao2026beyond,
title = {{Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks}},
author = {Hong Zhao},
howpublished = {arXiv},
year = {2026},
month = jul,
doi = {10.48550/arxiv.2607.08406},
eprint = {2607.08406},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.08406},
}
RIS
TY - UNPB
TI - Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
AU - Hong Zhao
PY - 2026
DA - 2026/07/09
PB - arXiv
DO - 10.48550/arxiv.2607.08406
UR - https://arxiv.org/abs/2607.08406
AB - Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients. The pursuit of gradient-free methods has long been a goal in the field of artificial intelligence. This paper shows that indeed the simplest Monte Carlo algorithm implemented on a single GPU -- randomly mutate a parameter, keep it if the loss decreases, otherwise retry -- can practically train deep networks. This gradient-free method does not even need common techniques such as batch normalization or residual connections to directly train sufficiently deep networks. More remarkably, its flexibility extends to several nontrivial scenarios: it enables pure pruning training, supports discrete weights, accommodates unconventional transfer functions such as Gaussian, and reveals the substantial redundancy of deep networks. We have demonstrated its feasibility on deep networks with more than 20 layers, single-hidden-layer wide networks with up to 16,384 hidden neurons, and even a simple Transformer architecture trained on both image classification (MNIST) and character-level language modeling (Tiny Shakespeare). This simple gradient-free method may offer a complementary perspective for understanding the self-organization and learning mechanisms of neural networks, and also provides an alternative route for building physically inspired deep learning systems.
ER -