Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent
Abstract
Pretraining followed by fine-tuning introduces a compute-allocation problem: under a fixed training budget, compute spent improving the upstream objective reduces the compute available for downstream adaptation. Despite its practical importance, this trade-off is not yet well understood theoretically, even in simple models. In this paper, we cast this allocation as a compute-split problem under a two-stage pretrain--fine-tune procedure with fixed total optimisation budget, using regularised least squares trained by gradient descent as a tractable setting. We characterise the optimal split under data-dependent evaluation geometries induced by the fine-tuning problem. Our results show that the allocation depends on how pretraining directions affect fine-tuning predictions and how fine-tuning shifts are seen through downstream data geometry. In particular, the relevant quantities are determined by prediction-relevant spectral components of the pretraining and fine-tuning empirical covariances. Technically, the analysis relies on a basis-invariant, eigenspace-level spectral decomposition, together with perturbative control of the non-commuting pretraining and fine-tuning dynamics.
Related
- Silver Rate Is (Almost) Optimal for Gradient Descent: The Strongly Convex Case
- Silver Rate Is (Almost) Optimal for Gradient Descent
- Exact Theory and Optimal Stepsizes for Stochastic Gradient Descent in Smooth Convex Minimization
- Optimal Recursive Composition and Dyadic Phase Laws for Gradient Descent with Predetermined Stepsizes
- Broximal Gradient Descent: A Projection-Free Sister of Projected Gradient Descent
BibTeX
@misc{buna2026computeoptimal,
title = {{Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent}},
author = {Alex Buna and Fanghui Liu and Patrick Rebeschini},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.16262},
eprint = {2609.16262},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.16262},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent
AU - Alex Buna
AU - Fanghui Liu
AU - Patrick Rebeschini
PY - 2026
DA - 2026/09/14
PB - arXiv
DO - 10.48550/arxiv.2609.16262
UR - https://arxiv.org/abs/2609.16262
AB - Pretraining followed by fine-tuning introduces a compute-allocation problem: under a fixed training budget, compute spent improving the upstream objective reduces the compute available for downstream adaptation. Despite its practical importance, this trade-off is not yet well understood theoretically, even in simple models. In this paper, we cast this allocation as a compute-split problem under a two-stage pretrain--fine-tune procedure with fixed total optimisation budget, using regularised least squares trained by gradient descent as a tractable setting. We characterise the optimal split under data-dependent evaluation geometries induced by the fine-tuning problem. Our results show that the allocation depends on how pretraining directions affect fine-tuning predictions and how fine-tuning shifts are seen through downstream data geometry. In particular, the relevant quantities are determined by prediction-relevant spectral components of the pretraining and fine-tuning empirical covariances. Technically, the analysis relies on a basis-invariant, eigenspace-level spectral decomposition, together with perturbative control of the non-commuting pretraining and fine-tuning dynamics.
ER -