High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning
Abstract
To commit to buying external data or participate in collaborative learning, one must decide whether the additional data will improve prediction enough to justify the cost. This comes with several challenges: (i) the decision often relies only on aggregated statistics available publicly, rather than individual-level data; (ii) covariate and model shifts can induce negative transfer, so the additional data deteriorates rather than improves performance; (iii) if the data is sensitive, its privatization requires the injection of noise, which can also offset the benefit of a larger sample size. In this paper, we model the problem of dataset selection through high-dimensional regression with multiple heterogeneous sources and a weighted ridge estimator. Our approach uses only summary statistics and it gives privacy guarantees either on labels only or jointly on features and labels, in terms of ρ-zero-concentrated differential privacy. The main technical contribution is a deterministic equivalent of the test error, which captures the interactions between sample size, covariance structure, model shift, regularization and privacy noise. Our theory allows to optimize hyperparameters (weights and ridge regularizers) and, more broadly, to decide when private external datasets are useful without accessing the data itself but only relying on population-level quantities. This provides a theoretically tractable foundation for private transfer learning, which we support via experiments on both synthetic and real-world datasets.
Related
- Beyond Source-Level Transfer: Sample-Level Learning for High-Dimensional Quantile Regression
- Joint-Sparse Transfer Learning for High-Dimensional Multi-Output Regression
- Finite- and long-range spin-wave free-energy asymptotics for the two-dimensional quantum Heisenberg ferromagnet
- Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics
- Normal and Lognormal Asymptotics for Lattice-Point Counts in Random Translates of High-Dimensional Balls
BibTeX
@misc{kovacevic2026highdimensional,
title = {{High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning}},
author = {Filip Kovačević and Edwige Cyffers and Stefano Sarao Mannelli and Marco Mondelli},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02578},
eprint = {2610.02578},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02578},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning
AU - Filip Kovačević
AU - Edwige Cyffers
AU - Stefano Sarao Mannelli
AU - Marco Mondelli
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.02578
UR - https://arxiv.org/abs/2610.02578
AB - To commit to buying external data or participate in collaborative learning, one must decide whether the additional data will improve prediction enough to justify the cost. This comes with several challenges: (i) the decision often relies only on aggregated statistics available publicly, rather than individual-level data; (ii) covariate and model shifts can induce negative transfer, so the additional data deteriorates rather than improves performance; (iii) if the data is sensitive, its privatization requires the injection of noise, which can also offset the benefit of a larger sample size. In this paper, we model the problem of dataset selection through high-dimensional regression with multiple heterogeneous sources and a weighted ridge estimator. Our approach uses only summary statistics and it gives privacy guarantees either on labels only or jointly on features and labels, in terms of ρ-zero-concentrated differential privacy. The main technical contribution is a deterministic equivalent of the test error, which captures the interactions between sample size, covariance structure, model shift, regularization and privacy noise. Our theory allows to optimize hyperparameters (weights and ridge regularizers) and, more broadly, to decide when private external datasets are useful without accessing the data itself but only relying on population-level quantities. This provides a theoretically tractable foundation for private transfer learning, which we support via experiments on both synthetic and real-world datasets.
ER -