Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
Abstract
We study the identity straight-through estimator (STE) for training a two-layer binary-activation network with hinge loss from the perspective of Statistical Learning Theory (SLT). Our central question is whether algorithmic stability can explain the statistical generalization of the estimator produced by the discontinuous STE training rule. In the saturated-output regime, the zero-initialized samplewise STE recursion is exactly the stochastic subgradient descent on the convex latent loss (−yu⊤ x)+. This representation makes a stability analysis possible. We derive an exact distance identity for two coupled updates and prove approximate non-expansiveness of the common-example map, with a quadratic defect only when the two latent margins straddle zero. We then obtain explicit ℓ2 on-average model-stability and generalization bounds, transferring stability isometrically from the latent vector to the full first-layer matrix. Combining stability with a standard optimization bound yields an explicit excess induced-risk guarantee and the rate O(n−1/2) when T=n2. Under margin separability, a complementary argument gives the optimal-order O(R2/(γ2n)) expected excess misclassification error for a randomized one-pass STE iterate and a corresponding majority-vote bound.
Related
- Beyond Shadow Weights: Quantization-Aware Training as Quantized-Endpoint Descent
- Harmonizable Multifractional Stable Motion: estimators for the Hurst function and the stability parameter from discrete-time observations, their rates of convergence and the associated central limit theorems
- Certifiably Interpretable Training of ReLU-MLPs for Boolean Tasks with Guaranteed Truth-Table Generalization
- On the Stability and Convergence of Flow Map Training
- A Sharp Bound of Crossing Limit Cycle for Planar piecewise linear Systems with a Straight Switching Line
BibTeX
@misc{ying2026stability,
title = {{Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks}},
author = {Yiming Ying},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.06430},
eprint = {2609.06430},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.06430},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
AU - Yiming Ying
PY - 2026
DA - 2026/09/06
PB - arXiv
DO - 10.48550/arxiv.2609.06430
UR - https://arxiv.org/abs/2609.06430
AB - We study the identity straight-through estimator (STE) for training a two-layer binary-activation network with hinge loss from the perspective of Statistical Learning Theory (SLT). Our central question is whether algorithmic stability can explain the statistical generalization of the estimator produced by the discontinuous STE training rule. In the saturated-output regime, the zero-initialized samplewise STE recursion is exactly the stochastic subgradient descent on the convex latent loss (−yu^⊤ x)_+. This representation makes a stability analysis possible. We derive an exact distance identity for two coupled updates and prove approximate non-expansiveness of the common-example map, with a quadratic defect only when the two latent margins straddle zero. We then obtain explicit ℓ_2 on-average model-stability and generalization bounds, transferring stability isometrically from the latent vector to the full first-layer matrix. Combining stability with a standard optimization bound yields an explicit excess induced-risk guarantee and the rate O(n^−1/2) when T=n^2. Under margin separability, a complementary argument gives the optimal-order O(R^2/(γ^2n)) expected excess misclassification error for a randomized one-pass STE iterate and a corresponding majority-vote bound.
ER -