PreprintCC BY 4.0arXiv
Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos
Abstract
We ask whether the standard treatment of dropout as a static hyperparameter is optimal, or whether its utility can be improved by letting it vary over depth. We answer this by developing a mean-field theory of dropout near the edge of chaos, identifying distinct universality classes for smooth and kinked activations, together with their scaling exponents. The resulting propagation theory, constrained by maximizing the regularization delivered by dropout, motivates concentrating dropout near the input. Experiments on vision, speech and financial time series show gains most consistently in MLPs, with smaller gains in Transformers.
Related
- Bulk universality of scaling limits for the almost Hermitian Coulomb gas
- Optimal Transport Dropout for Structured Predictive Uncertainty
- Mean-field branching SDEs: propagation of chaos, scaling limits and phase transitions
- Directed distances in spanning-tree-decorated planar maps: exact exponent, scaling limit and universality
- Optimal Uniform-like Scheduling of Maintenance
BibTeX
@misc{sarmiento2026dropout,
title = {{Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos}},
author = {Lucas Fernandez Sarmiento},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.21648},
eprint = {2605.21648},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.21648},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos
AU - Lucas Fernandez Sarmiento
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2605.21648
UR - https://arxiv.org/abs/2605.21648
AB - We ask whether the standard treatment of dropout as a static hyperparameter is optimal, or whether its utility can be improved by letting it vary over depth. We answer this by developing a mean-field theory of dropout near the edge of chaos, identifying distinct universality classes for smooth and kinked activations, together with their scaling exponents. The resulting propagation theory, constrained by maximizing the regularization delivered by dropout, motivates concentrating dropout near the input. Experiments on vision, speech and financial time series show gains most consistently in MLPs, with smaller gains in Transformers.
ER -