Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
Abstract
Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained \emph{source} model is trained via ridge regression, and (ii) a fine-tuned \emph{target} model is regularized via the L2 distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.
Related
- Optimal Tradeoffs Between Network Size and Parameter Magnitude in Neural Approximation and Minimax Regression
- Minimax and Adaptive Transfer Learning for Sparse Canonical Correlation Analysis
- Is Spectral Self-Supervised Learning a Minimax Optimal Method of Density Ratio Estimation?
- Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers
- SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws
BibTeX
@misc{shabanzadeh2026minimaxoptimal,
title = {{Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning}},
author = {Yalda Shabanzadeh and Diyuan Wu and Marco Mondelli},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.23337},
eprint = {2610.23337},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.23337},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
AU - Yalda Shabanzadeh
AU - Diyuan Wu
AU - Marco Mondelli
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2610.23337
UR - https://arxiv.org/abs/2610.23337
AB - Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained \emph{source} model is trained via ridge regression, and (ii) a fine-tuned \emph{target} model is regularized via the L_2 distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.
ER -