An Active-Bottleneck Mechanism for Weak-to-Strong Generalization
Abstract
Weak-to-strong generalization (W2SG) occurs when a student trained on a teacher's predictions outperforms that teacher. We study when this happens under fully converged, ridgeless two-stage learning, with no early stopping, no explicit regularization, and no assumption that the student is more expressive than the teacher. In two-stage linear regression, a teacher is fit from n labeled examples and a student is trained solely on the teacher's predictions on m fresh, unlabeled inputs. Although both stages share the same hypothesis class and the same training rule, we show that the student outperforms the teacher exactly when m lies in an explicit intermediate range: too few pseudo-labels leave the student without enough signal, too many let it inherit the teacher's noise. Under power-law covariance, we derive this range in closed form as a function of the spectral decay and noise level, including regimes where the improving region splits into two disjoint intervals of m. We then study a random-feature model in which the student has strictly more features than the teacher, and identify two regimes, again given by explicit thresholds: one where improvement occurs only for m in a bounded interval, and one where it occurs only once the student width NS exceeds an explicit threshold. Both regimes are governed by a single "active-bottleneck" principle: whichever of m or NS is scarcer controls how much teacher error is filtered out, while increasing the other resource only reduces estimation noise. Together, these results show that finite data and finite width can themselves regularize a two-stage learner, with no explicit mechanism doing so.
Related
- Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory
- Weak and strong Lefschetz properties for vertex cover Artinian algebras associated to graphs
- Weak-strong uniqueness for the Landau equation by a relative entropy method
- Weak and strong convergence of a general splitting approach for monotone multioperator inclusions
- Strong and Weak-Type Dispersive Estimates for the Energy-Critical Nonlinear Schrödinger Equation with an Inverse-Square Potential
BibTeX
@misc{zeinalpour2026activebottleneck,
title = {{An Active-Bottleneck Mechanism for Weak-to-Strong Generalization}},
author = {Mohammad Zeinalpour and Amir Najafi},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.33835},
eprint = {2609.33835},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.33835},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - An Active-Bottleneck Mechanism for Weak-to-Strong Generalization
AU - Mohammad Zeinalpour
AU - Amir Najafi
PY - 2026
DA - 2026/09/27
PB - arXiv
DO - 10.48550/arxiv.2609.33835
UR - https://arxiv.org/abs/2609.33835
AB - Weak-to-strong generalization (W2SG) occurs when a student trained on a teacher's predictions outperforms that teacher. We study when this happens under fully converged, ridgeless two-stage learning, with no early stopping, no explicit regularization, and no assumption that the student is more expressive than the teacher. In two-stage linear regression, a teacher is fit from n labeled examples and a student is trained solely on the teacher's predictions on m fresh, unlabeled inputs. Although both stages share the same hypothesis class and the same training rule, we show that the student outperforms the teacher exactly when m lies in an explicit intermediate range: too few pseudo-labels leave the student without enough signal, too many let it inherit the teacher's noise. Under power-law covariance, we derive this range in closed form as a function of the spectral decay and noise level, including regimes where the improving region splits into two disjoint intervals of m. We then study a random-feature model in which the student has strictly more features than the teacher, and identify two regimes, again given by explicit thresholds: one where improvement occurs only for m in a bounded interval, and one where it occurs only once the student width N_S exceeds an explicit threshold. Both regimes are governed by a single "active-bottleneck" principle: whichever of m or N_S is scarcer controls how much teacher error is filtered out, while increasing the other resource only reduces estimation noise. Together, these results show that finite data and finite width can themselves regularize a two-stage learner, with no explicit mechanism doing so.
ER -