Feature Normalization under Finite Training Budgets
Abstract
Standardizing a rare feature can help stochastic gradient descent learn its effect within a finite budget, but it also changes the noise cost of learning. We characterize the lowest rare-subgroup risks attainable by standardized and unscaled training when each chooses a nonnegative constant learning rate and a deterministic stopping time under the same observation cap. For two independent binary features, we derive both optimized risks as prevalence vanishes while the expected number of rare observations remains finite. Uniform control over rates and stopping times reduces each limiting oracle to a scalar minimization. With positive noise, the preferred scaling changes exactly once as we move signal to the rare feature while the limiting risk at initialization stays fixed and positive. The formulas also explain how noise and budget affect the comparison. Separate finite-prevalence bounds favor standardization on a rare-dominant signal family and unscaled training on a common-dominant family. The results concern final iterates with population preprocessing and fresh observations. Exact-moment searches assess the approximation at finite prevalence. In a controlled study, both orderings persist under observation reuse, estimated preprocessing, and intercept fitting when tuning choices are fixed before evaluation. Selecting a predictor after observing training can reverse an ordering, even when selection uses exact population risk.
Related
- Coordinate descents, monodromy, and finite normalization of marked-root maps
- Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
- Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines
- Joint Estimation and Feature Selection in Overspecified Finite Mixture of Regressions
- A Localized Hybrid Finite Element--Random Feature Method for Elliptic Interface Problems on Unfitted Meshes
BibTeX
@misc{larsson2026feature,
title = {{Feature Normalization under Finite Training Budgets}},
author = {Johan Larsson},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.16845},
eprint = {2610.16845},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.16845},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Feature Normalization under Finite Training Budgets
AU - Johan Larsson
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2610.16845
UR - https://arxiv.org/abs/2610.16845
AB - Standardizing a rare feature can help stochastic gradient descent learn its effect within a finite budget, but it also changes the noise cost of learning. We characterize the lowest rare-subgroup risks attainable by standardized and unscaled training when each chooses a nonnegative constant learning rate and a deterministic stopping time under the same observation cap. For two independent binary features, we derive both optimized risks as prevalence vanishes while the expected number of rare observations remains finite. Uniform control over rates and stopping times reduces each limiting oracle to a scalar minimization. With positive noise, the preferred scaling changes exactly once as we move signal to the rare feature while the limiting risk at initialization stays fixed and positive. The formulas also explain how noise and budget affect the comparison. Separate finite-prevalence bounds favor standardization on a rare-dominant signal family and unscaled training on a common-dominant family. The results concern final iterates with population preprocessing and fresh observations. Exact-moment searches assess the approximation at finite prevalence. In a controlled study, both orderings persist under observation reuse, estimated preprocessing, and intercept fitting when tuning choices are fixed before evaluation. Selecting a predictor after observing training can reverse an ordering, even when selection uses exact population risk.
ER -