Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes
Abstract
Classical PAC-Bayesian generalization theory places prior distributions over Euclidean parameter space Θ⊆ ℝD, causing relative entropy to scale with network dimension D. For modern overparameterized neural networks where D ≫ n, this parameter-counting dependency yields vacuous generalization bounds (> 1,000%). In this work, we resolve this barrier by shifting prior specification from internal parameters to the non-parametric space of data distributions ℳ1(𝒵) via a \textbf{Dirichlet Process (DP)}, 𝒟𝒫(α, F0), inducing parameter distributions through push-forward measures under the learning algorithm. By the Data Processing Inequality, parameter-space relative entropy is strictly bounded by the data-space DP divergence, which we evaluate in exact closed form using Digamma functions. The resulting PAC-Bayesian certificate is strictly invariant to network parameter count D, width, and depth. On intrinsic d*-dimensional submanifolds, optimal partition scaling achieves the minimax non-parametric rate 𝒪(n−1/(d*+2) √(ln n)). Across vision architectures scaling up to 4.2 million parameters (D = 7,850 to D = 4,203,522) on the full MNIST benchmark, while classical weight-space bounds scale past 940% (becoming vacuous), our data-space DP-PAC-Bayes certificate remains stable, non-vacuous, and certified at \textbf{8.8%--9.4%}.
Related
- Pointwise Generalization in Deep Neural Networks
- A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
- Bayesian Neural Networks via Dirichlet Process Priors on the Data-Generating Distribution for Randomized Value Functions in Deep Reinforcement Learning
- Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
- Emergent Generalization by Representation Learning in Artificial Neural Networks
BibTeX
@misc{vashishtha2026dimensionfree,
title = {{Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes}},
author = {Sumit Vashishtha},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.21340},
eprint = {2610.21340},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.21340},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes
AU - Sumit Vashishtha
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.21340
UR - https://arxiv.org/abs/2610.21340
AB - Classical PAC-Bayesian generalization theory places prior distributions over Euclidean parameter space Θ⊆ ℝ^D, causing relative entropy to scale with network dimension D. For modern overparameterized neural networks where D ≫ n, this parameter-counting dependency yields vacuous generalization bounds (> 1,000%). In this work, we resolve this barrier by shifting prior specification from internal parameters to the non-parametric space of data distributions ℳ_1(𝒵) via a \textbf{Dirichlet Process (DP)}, 𝒟𝒫(α, F_0), inducing parameter distributions through push-forward measures under the learning algorithm. By the Data Processing Inequality, parameter-space relative entropy is strictly bounded by the data-space DP divergence, which we evaluate in exact closed form using Digamma functions. The resulting PAC-Bayesian certificate is strictly invariant to network parameter count D, width, and depth. On intrinsic d^*-dimensional submanifolds, optimal partition scaling achieves the minimax non-parametric rate 𝒪(n^−1/(d^*+2) √(ln n)). Across vision architectures scaling up to 4.2 million parameters (D = 7,850 to D = 4,203,522) on the full MNIST benchmark, while classical weight-space bounds scale past 940% (becoming vacuous), our data-space DP-PAC-Bayes certificate remains stable, non-vacuous, and certified at \textbf{8.8%--9.4%}.
ER -