Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features

Abstract

We study the last iterate of stochastic gradient descent (SGD) on streaming least squares with discretely distributed features (Slock's model), which decouples the dynamics across eigendirections. Under power-law source and capacity conditions, we derive the exact finite-time risk and asymptotic equivalents of constant and Warmup-Stable-Decay (WSD) schedules. With a tuned base learning-rate, the resulting scaling law decays as Θ(T^−q), where T is the horizon, and q is independent of the cooldown length c ∈ (0,1] but is strictly smaller for the constant schedule: any cooldown of fixed relative length c > 0, however short, improves the risk by a polynomial factor, while its length only affects the constant, which is minimized by linear decay (c = 1). Finally, we show that the optimal risk depends on the batch size b and horizon T only through bT, up to a critical batch size.

Distributed online learning with growing dimensions for stochastic large regression
Die Gan, Zhixin Liu
arXiv29 Oct 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
On asymptotic expansions of the density of states for Poisson distributed random Schrödinger operators
Jan Fischer, David Hasler, Jannis Koberstein
arXiv10 Sep 2026PreprintCC BY-NC-SA 4.0Full textMathematical physicsarXiv
Towards optimal selection schedules for online active learning
Hugo Cui, Francesca Mignacco, Francesco Mori
arXiv16 Oct 2026PreprintCC BY 4.0Full textStatistics Condensed matterarXiv
Clouded Representatives and Score Matching for Distributed Quantile Regression
Keren Li, Boxiang Wang
arXiv5 Nov 2026PreprintCC BY 4.0Full textStatisticsarXiv
Distributed Selective Inference for Quantile Regression
Xiaohui Yuan, Jiahan Teng, Yan Zhou
arXiv12 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv

Cite

BibTeX RIS

BibTeX

@misc{colomban2026asymptotic,
  title = {{Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features}},
  author = {Noé Colomban and Francis Bach and Fabian Schaipp},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.16518},
  eprint = {2610.16518},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.16518},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features
AU  - Noé Colomban
AU  - Francis Bach
AU  - Fabian Schaipp
PY  - 2026
DA  - 2026/10/12
PB  - arXiv
DO  - 10.48550/arxiv.2610.16518
UR  - https://arxiv.org/abs/2610.16518
AB  - We study the last iterate of stochastic gradient descent (SGD) on streaming least squares with discretely distributed features (Slock's model), which decouples the dynamics across eigendirections. Under power-law source and capacity conditions, we derive the exact finite-time risk and asymptotic equivalents of constant and Warmup-Stable-Decay (WSD) schedules. With a tuned base learning-rate, the resulting scaling law decays as Θ(T^−q), where T is the horizon, and q is independent of the cooldown length c ∈ (0,1] but is strictly smaller for the constant schedule: any cooldown of fixed relative length c > 0, however short, improves the risk by a polynomial factor, while its length only affects the constant, which is minimized by linear decay (c = 1). Finally, we show that the optimal risk depends on the batch size b and horizon T only through bT, up to a critical batch size.
ER  -

Abstract

Related

Cite