From Dataset Spectral Geometry to Network Weights: A Geometry-Aware Initialization for Sigmoidal MLPs in Image Classification
Abstract
Classical universal approximation theorems (UAT) establish the expressive power of sigmoidal multilayer perceptrons, but they do not specify how the weights should be initialized. We study a supervised, data-dependent, geometry-aware initialization for one-hidden-layer sigmoidal MLPs that compiles labeled class geometry into network weights. The construction starts from the idea that sigmoid units can act as smooth half-space gates. For each class, we center the training samples at their mean, apply SVD to estimate principal directions and spectral scales, select retained directions by an energy threshold, and represent each retained direction by a pair of sigmoid slab gates. These class-specific gates are then concatenated into a shared hidden layer initialized directly from the training set. We also formulate a SVD-Mahalanobis subspace classifier as a non-neural geometric reference, which tests whether the estimated spectral class geometry is already discriminative before being embedded into the MLP. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 show that the proposed initializer produces a substantially more informative zero-epoch state than a matched task-agnostic Xavier reference, while full training reaches comparable final accuracy. Frozen-hidden experiments and neutral-head ablations further show that the class-wise SVD gates remain useful fixed features, even after the initially aligned output layer is replaced by a Xavier-initialized one.
Related
- Comparative Classification of DNS-over-HTTPS Traffic on the CIRACIC-DoHBrw-2020 Dataset Using Logistic Regression, Random Forest and XGBoost
- Benchmarking Classical Machine Learning and Deep Learning Models on the Cleveland Heart Disease Binary Classification Dataset
- Handwriting-based Alzheimer’s disease classification on the DARWIN dataset: Repeated nested validation, task-level reduction, and cross-validated explainability
- Centro-affine spectral geometry of polytopes
- Spectral geometry: old questions and new answers
BibTeX
@misc{chu2026from,
title = {{From Dataset Spectral Geometry to Network Weights: A Geometry-Aware Initialization for Sigmoidal MLPs in Image Classification}},
author = {Yi-Shan Chu},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.28444},
eprint = {2606.28444},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.28444},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - From Dataset Spectral Geometry to Network Weights: A Geometry-Aware Initialization for Sigmoidal MLPs in Image Classification
AU - Yi-Shan Chu
PY - 2026
DA - 2026/09/30
PB - arXiv
DO - 10.48550/arxiv.2606.28444
UR - https://arxiv.org/abs/2606.28444
AB - Classical universal approximation theorems (UAT) establish the expressive power of sigmoidal multilayer perceptrons, but they do not specify how the weights should be initialized. We study a supervised, data-dependent, geometry-aware initialization for one-hidden-layer sigmoidal MLPs that compiles labeled class geometry into network weights. The construction starts from the idea that sigmoid units can act as smooth half-space gates. For each class, we center the training samples at their mean, apply SVD to estimate principal directions and spectral scales, select retained directions by an energy threshold, and represent each retained direction by a pair of sigmoid slab gates. These class-specific gates are then concatenated into a shared hidden layer initialized directly from the training set. We also formulate a SVD-Mahalanobis subspace classifier as a non-neural geometric reference, which tests whether the estimated spectral class geometry is already discriminative before being embedded into the MLP. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 show that the proposed initializer produces a substantially more informative zero-epoch state than a matched task-agnostic Xavier reference, while full training reaches comparable final accuracy. Frozen-hidden experiments and neutral-head ablations further show that the class-wise SVD gates remain useful fixed features, even after the initially aligned output layer is replaced by a Xavier-initialized one.
ER -