On the robustness of noisy solutions in non-convex neural networks
Abstract
Optimization in non-convex neural network models is strongly influenced by the geometry of the solution space: sparse, isolated, point-like clusters are typically algorithmically inaccessible, whereas wide and flat regions can be found efficiently despite being relatively rare. At zero temperature this picture has been formalized in binary perceptrons through the overlap gap property (OGP), which limits algorithmic access to configurations with zero training error above a critical constraint density αOGP. Here we extend this description to finite temperature, where a positive training error is allowed and statistically penalized. We first show that the frozen one-step replica-symmetry-breaking solution, dominating the zero temperature equilibrium measure, survives at any finite temperature. We furthermore derive a general criterion, based on the smoothness of the single-pattern Gibbs weight near the decision boundary, that determines when a finite-temperature relaxation of the loss removes freezing. We then extend the OGP construction to finite temperature and show that dense, algorithmically accessible regions of finite-energy configurations persist beyond αOGP, up to a threshold αOGP(ε) that grows with the allowed training error ε. Finally, in the teacher-student setting, we show that these wide, finite-energy regions still retain good generalization. Using a finite energy message-passing algorithm, we demonstrate numerically that thermal noise enables effective generalization in the regime of constraint densities where both recovering the teacher and finding a zero temperature solution are computationally hard.
Related
- Exact Convex Reformulations of Linear Neural Networks via Completely Positive Lifting
- Dual Certified White-Box Inference for Input Convex Neural Networks
- Approximation of solutions of parameter-dependent problems by residual neural networks
- AKAPINN: Adaptive Kolmogorov-Arnold Physics-Informed Neural Networks for approximating solutions to quasilinear partial differential equations
- Optimization-based lane-changing trajectory planning in mixed traffic with embedded input convex neural networks
BibTeX
@misc{malatesta2026robustness,
title = {{On the robustness of noisy solutions in non-convex neural networks}},
author = {Enrico M. Malatesta and Alessandra Passalacqua and Riccardo Zecchina},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2607.27000},
eprint = {2607.27000},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.27000},
}
RIS
TY - UNPB
TI - On the robustness of noisy solutions in non-convex neural networks
AU - Enrico M. Malatesta
AU - Alessandra Passalacqua
AU - Riccardo Zecchina
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2607.27000
UR - https://arxiv.org/abs/2607.27000
AB - Optimization in non-convex neural network models is strongly influenced by the geometry of the solution space: sparse, isolated, point-like clusters are typically algorithmically inaccessible, whereas wide and flat regions can be found efficiently despite being relatively rare. At zero temperature this picture has been formalized in binary perceptrons through the overlap gap property (OGP), which limits algorithmic access to configurations with zero training error above a critical constraint density α_OGP. Here we extend this description to finite temperature, where a positive training error is allowed and statistically penalized. We first show that the frozen one-step replica-symmetry-breaking solution, dominating the zero temperature equilibrium measure, survives at any finite temperature. We furthermore derive a general criterion, based on the smoothness of the single-pattern Gibbs weight near the decision boundary, that determines when a finite-temperature relaxation of the loss removes freezing. We then extend the OGP construction to finite temperature and show that dense, algorithmically accessible regions of finite-energy configurations persist beyond α_OGP, up to a threshold α_OGP(ε) that grows with the allowed training error ε. Finally, in the teacher-student setting, we show that these wide, finite-energy regions still retain good generalization. Using a finite energy message-passing algorithm, we demonstrate numerically that thermal noise enables effective generalization in the regime of constraint densities where both recovering the teacher and finding a zero temperature solution are computationally hard.
ER -