When is early stopping beneficial under label noise? A phase diagram for kernel gradient flow
Abstract
Early stopping is commonly used to prevent fitting label noise from degrading generalization. Does this mean that, when a positive fraction of the training labels is intrinsically unpredictable, the best test error must occur before interpolation? Experiments on a CIFAR10 classification task with randomly labeled examples motivate the opposite possibility: although fixed-feature and shallow models exhibit late-time deterioration of the test error, the need for early stopping essentially disappears in larger, deeper convolutional networks even after the random labels have all been classified correctly. We determine when late-time deterioration disappears entirely in a solvable setting. For kernel gradient flow on a synthetic noisy classification task, we derive sharp high-dimensional learning curves and obtain a phase diagram controlled by the balance between the linear kernel component that learns the shared relation between inputs and labels and the higher-order components that enable fitting the label noise. In one regime, generalization is optimized at finite time and early stopping is necessary. In another, generalization improves until interpolation, but the endpoint is suboptimal across kernels. At the boundary, the interpolating endpoint attains the lowest generalization error across the considered kernel family and all training times. Thus, even with extensive irreducible label noise, interpolation can outperform early stopping. The neural-network experiments leave open how learned representations acquire an analogous favorable geometry.
Related
- Early stopping of stochastic variance reduced gradient for linear inverse problems by the discrepancy principle
- Sequential Automorphism Ensemble Decoding with Early Stopping
- Gradient estimates for the Green kernel under spectral Ricci bounds, and the stable Bernstein theorem in ℝ⁴
- Minimax-Optimal Early Stopping for Continuous-Time SGD via the Discrepancy Principle
- A functional central limit theorem for kernel gradient flow and infinitesimal gradient boosting
BibTeX
@misc{boncoraglio2026when,
title = {{When is early stopping beneficial under label noise? A phase diagram for kernel gradient flow}},
author = {Fabrizio Boncoraglio and Lenka Zdeborová},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.33846},
eprint = {2610.33846},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33846},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - When is early stopping beneficial under label noise? A phase diagram for kernel gradient flow
AU - Fabrizio Boncoraglio
AU - Lenka Zdeborová
PY - 2026
DA - 2026/10/24
PB - arXiv
DO - 10.48550/arxiv.2610.33846
UR - https://arxiv.org/abs/2610.33846
AB - Early stopping is commonly used to prevent fitting label noise from degrading generalization. Does this mean that, when a positive fraction of the training labels is intrinsically unpredictable, the best test error must occur before interpolation? Experiments on a CIFAR10 classification task with randomly labeled examples motivate the opposite possibility: although fixed-feature and shallow models exhibit late-time deterioration of the test error, the need for early stopping essentially disappears in larger, deeper convolutional networks even after the random labels have all been classified correctly. We determine when late-time deterioration disappears entirely in a solvable setting. For kernel gradient flow on a synthetic noisy classification task, we derive sharp high-dimensional learning curves and obtain a phase diagram controlled by the balance between the linear kernel component that learns the shared relation between inputs and labels and the higher-order components that enable fitting the label noise. In one regime, generalization is optimized at finite time and early stopping is necessary. In another, generalization improves until interpolation, but the endpoint is suboptimal across kernels. At the boundary, the interpolating endpoint attains the lowest generalization error across the considered kernel family and all training times. Thus, even with extensive irreducible label noise, interpolation can outperform early stopping. The neural-network experiments leave open how learned representations acquire an analogous favorable geometry.
ER -