Pointwise Generalization in Deep Neural Networks
Abstract
We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundation for representation learning. For each trained model, we characterize the hypothesis via a pointwise Riemannian Dimension, derived from the eigenvalues of the learned feature representations across layers. This establishes a principled framework for deriving hypothesis-dependent, representation-aware generalization bounds. These bounds offer a systematic upgrade over approaches based on model size, products of norms, and infinite-width linearizations, yielding guarantees that are orders of magnitude tighter in both theory and experiment. Analytically, we identify the structural properties and mathematical principles that explain the tractability of deep networks. Empirically, the pointwise Riemannian Dimension exhibits substantial feature compression, decreases with increased over-parameterization, and captures the implicit bias of optimizers. Taken together, our results indicate that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, feature-spectrum-aware complexity.
Related
- Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes
- A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
- Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
- Emergent Generalization by Representation Learning in Artificial Neural Networks
- Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention
BibTeX
@misc{li2026pointwise,
title = {{Pointwise Generalization in Deep Neural Networks}},
author = {Shaojie Li and Yunbei Xu},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.18598},
eprint = {2605.18598},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.18598},
}
RIS
TY - UNPB
TI - Pointwise Generalization in Deep Neural Networks
AU - Shaojie Li
AU - Yunbei Xu
PY - 2026
DA - 2026/09/24
PB - arXiv
DO - 10.48550/arxiv.2605.18598
UR - https://arxiv.org/abs/2605.18598
AB - We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundation for representation learning. For each trained model, we characterize the hypothesis via a pointwise Riemannian Dimension, derived from the eigenvalues of the learned feature representations across layers. This establishes a principled framework for deriving hypothesis-dependent, representation-aware generalization bounds. These bounds offer a systematic upgrade over approaches based on model size, products of norms, and infinite-width linearizations, yielding guarantees that are orders of magnitude tighter in both theory and experiment. Analytically, we identify the structural properties and mathematical principles that explain the tractability of deep networks. Empirically, the pointwise Riemannian Dimension exhibits substantial feature compression, decreases with increased over-parameterization, and captures the implicit bias of optimizers. Taken together, our results indicate that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, feature-spectrum-aware complexity.
ER -