Estimating Implicit Regularization in Deep Learning
Abstract
Deep learning systems are known to exhibit implicit regularization (alt. implicit bias), favoring simple solutions instead of merely minimizing the loss function. In some cases, we can analytically derive the implicit regularization -- connecting it to an equivalent penalty that augments the learning objective. However, modern deep learning systems are complex, carrying modifications to the training procedure and architecture (e.g. early stopping, minibatching, dropout) whose effects are not always directly interpretable. Although estimating the resulting implicit regularization could aid theorists in algorithm design and practitioners in interpreting their hyperparameter choices, this problem has received little direct attention. It is also tractable: regularization makes weight updates deviate from loss gradients, promising a signal for identifying implicit bias. Here we provide gradient matching methods that can be used to empirically estimate the implicit regularization. Our method works on networks with known regularization, recovering popular explicit penalties like ℓ1 and ℓ2. It also replicates known implicit effects, like the quadratic weight penalty induced by early stopping in gradient descent, demonstrating that it can be used to test theories of implicit regularization. Crucially, because our method is empirical, it can handle implicit regularization in arbitrary networks. We demonstrate this use by characterizing implicit ℓ2 effects of dropout in deep networks. Our work shows that practitioners can use gradient matching to understand regularization in networks with implicit biases that are too complicated to derive analytically.
Related
- Deep Learning for Reflected BSDEs: Regularization and Error Analysis
- DREAM: Deep-Reparametrization of Adaptive Regularization Maps for Fast Zero-Shot Self-Supervised Learning
- Strong convergence of a slowly damped primal-dual dynamical system with general time-scaling, Tikhonov regularization and implicit Hessian damping
- Convergence rates of mixed primal-dual dynamical systems with implicit Hessian damping and Tikhonov regularization
- From Good Starts to Optimal Inference: Generalized Latent Factor Models with Missingness and Implicit Regularization
BibTeX
@misc{rudoler2026estimating,
title = {{Estimating Implicit Regularization in Deep Learning}},
author = {Joseph H. Rudoler and Kevin Tan and Giles Hooker and Konrad P. Kording},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2605.05436},
eprint = {2605.05436},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.05436},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Estimating Implicit Regularization in Deep Learning
AU - Joseph H. Rudoler
AU - Kevin Tan
AU - Giles Hooker
AU - Konrad P. Kording
PY - 2026
DA - 2026/11/04
PB - arXiv
DO - 10.48550/arxiv.2605.05436
UR - https://arxiv.org/abs/2605.05436
AB - Deep learning systems are known to exhibit implicit regularization (alt. implicit bias), favoring simple solutions instead of merely minimizing the loss function. In some cases, we can analytically derive the implicit regularization -- connecting it to an equivalent penalty that augments the learning objective. However, modern deep learning systems are complex, carrying modifications to the training procedure and architecture (e.g. early stopping, minibatching, dropout) whose effects are not always directly interpretable. Although estimating the resulting implicit regularization could aid theorists in algorithm design and practitioners in interpreting their hyperparameter choices, this problem has received little direct attention. It is also tractable: regularization makes weight updates deviate from loss gradients, promising a signal for identifying implicit bias. Here we provide gradient matching methods that can be used to empirically estimate the implicit regularization. Our method works on networks with known regularization, recovering popular explicit penalties like ℓ_1 and ℓ_2. It also replicates known implicit effects, like the quadratic weight penalty induced by early stopping in gradient descent, demonstrating that it can be used to test theories of implicit regularization. Crucially, because our method is empirical, it can handle implicit regularization in arbitrary networks. We demonstrate this use by characterizing implicit ℓ_2 effects of dropout in deep networks. Our work shows that practitioners can use gradient matching to understand regularization in networks with implicit biases that are too complicated to derive analytically.
ER -