Beyond Affine Layers: Bounded Ball Coverings to Preclude Gradient Vanishing
Abstract
Traditional deep learning architectures relying on standard affine layers (WX +b) partition the activation space into unbounded polyhedral regions, which inherently introduce chronic training instabilities such as gradient vanishing and severe generalization decay. To resolve this structural vulnerability, we introduce an alternative “bounded-by-design” geometric framework that completely moves beyond traditional affine transformations. By replacing them entirely with ball coverings, we reformulate layer mechanics around the fundamental metric of a feature point’s proximity to a ball centered atW. Crucially, computing the formulation using direct dot products fully preserves GPU parallelization performance without altering external network components. Empirical validation demonstrates that our bounded formulation effectively precludes overfitting, maintains numerical stability, and exhibits superior testing generalization compared to unconstrained affine layers.
Related
- Constraint Qualifications and Gradient Flows for Block Vanishing Constraint Problems
- Gradient-Free Sampling from Generative Models via Stochastic Bounded Extremum Seeking
- Beyond Constant Truncations: The Bounded Sobolev Approximation Gap
- Minimum-degree thresholds beyond bounded colourability
- Spatial Mixing and Deterministic Approximate Counting of Multi-spin Systems beyond Bounded Degree Graphs
BibTeX
@misc{wu2026beyond,
title = {{Beyond Affine Layers: Bounded Ball Coverings to Preclude Gradient Vanishing}},
author = {Mengnien Wu},
howpublished = {HAL},
year = {2026},
month = sep,
url = {https://hal.science/hal-05742801v1},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Beyond Affine Layers: Bounded Ball Coverings to Preclude Gradient Vanishing
AU - Mengnien Wu
PY - 2026
DA - 2026/09/08
PB - HAL
UR - https://hal.science/hal-05742801v1
AB - Traditional deep learning architectures relying on standard affine layers (WX +b) partition the activation space into unbounded polyhedral regions, which inherently introduce chronic training instabilities such as gradient vanishing and severe generalization decay. To resolve this structural vulnerability, we introduce an alternative “bounded-by-design” geometric framework that completely moves beyond traditional affine transformations. By replacing them entirely with ball coverings, we reformulate layer mechanics around the fundamental metric of a feature point’s proximity to a ball centered atW. Crucially, computing the formulation using direct dot products fully preserves GPU parallelization performance without altering external network components. Empirical validation demonstrates that our bounded formulation effectively precludes overfitting, maintains numerical stability, and exhibits superior testing generalization compared to unconstrained affine layers.
ER -