A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
Abstract
Hard-label classification is usually trained with smooth surrogate losses such as softmax cross-entropy. We isolate a mechanism by which this mismatch between smooth surrogate and discrete labels produces power-law learning curves for a one-layer softmax student trained by online SGD on Gaussian inputs with argmax teacher labels. In the thermodynamic limit, the dynamics reduce to a growing student-teacher alignment and a residual student variance sustained by SGD noise. At late times, only shrinking layers of examples around teacher decision boundaries remain active. This yields an α−1/3 law in training time α not only for the test loss but also for the generalization error, one minus test accuracy. The error law is an online-noise effect. The loss law persists without gradient noise. Learning-rate annealing separates the two exponents, improving the error exponent from 1/3 toward 1/2 while slowing the loss. Simulations support the predictions. Correlated Gaussian inputs and whitened pretrained features change transients, not the late-time decay. Experiments with deep neural networks can show compatible error scaling. The mechanism is asymptotic and complementary to spectral explanations and data-statistics explanations of neural scaling laws.
Related
- Scaling of boundary-layer disturbances exposed to free-stream turbulence
- Physics-Constrained Symbolic Regression for Mean-Velocity and Mixing-Length Scaling in Zero-Pressure-Gradient Turbulent Boundary Layers
- Mean flow scaling in stably stratified temporally developing turbulent boundary layers
- Boundary-contact repolarization provides a mechanism for directional guidance of neural crest cell collectives
- Towards Scaling Quantum Fine-Tuning of Foundational Time Series Models for Classification
BibTeX
@misc{kuhn2026boundarylayer,
title = {{A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification}},
author = {Marcel Kühn and Yoon Thelge and Bernd Rosenow},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.22341},
eprint = {2605.22341},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.22341},
}
RIS
TY - UNPB
TI - A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
AU - Marcel Kühn
AU - Yoon Thelge
AU - Bernd Rosenow
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2605.22341
UR - https://arxiv.org/abs/2605.22341
AB - Hard-label classification is usually trained with smooth surrogate losses such as softmax cross-entropy. We isolate a mechanism by which this mismatch between smooth surrogate and discrete labels produces power-law learning curves for a one-layer softmax student trained by online SGD on Gaussian inputs with argmax teacher labels. In the thermodynamic limit, the dynamics reduce to a growing student-teacher alignment and a residual student variance sustained by SGD noise. At late times, only shrinking layers of examples around teacher decision boundaries remain active. This yields an α^−1/3 law in training time α not only for the test loss but also for the generalization error, one minus test accuracy. The error law is an online-noise effect. The loss law persists without gradient noise. Learning-rate annealing separates the two exponents, improving the error exponent from 1/3 toward 1/2 while slowing the loss. Simulations support the predictions. Correlated Gaussian inputs and whitened pretrained features change transients, not the late-time decay. Experiments with deep neural networks can show compatible error scaling. The mechanism is asymptotic and complementary to spectral explanations and data-statistics explanations of neural scaling laws.
ER -