Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysis
Abstract
Retrieval-Guided Fine-Tuning (RAG-FT) incorporates retrieved data directly into the training objective, but the statistical consequences of noisy retrieval during training remain theoretically undercharacterized. We study this question by modeling RAG-FT as an estimation problem in a multi-task linear regression framework, using an OLS proxy for single-layer linear self-attention to obtain finite-sample risk bounds. Under homoscedastic retrieval noise, we show that retrieval failure decays exponentially with task separation relative to noise, and derive explicit finite-sample conditions under which RAG-FT achieves lower risk than both target-only and full-corpus training. We then introduce a Distance-Proportional Noise (DPN) model, in which retrieval quality degrades with rank, and compare two estimators under the same retrieval process: the OLS proxy and the literal, uniform-weight forward pass of linear self-attention. We prove that the attention estimator's bias diverges as Θ(n2q) even under exact retrieval, while OLS risk remains Θ(d/n) for every noise exponent q>0. These results locate the instability not in noisy retrieval itself, but in the fixed, unweighted aggregation of the literal LSA forward pass, which reweighting by reliability empirically removes. We validate the predicted rate separation through direct simulation of the DPN model.
Related
- Energy-Guided Learning for Heat Equation with Noisy Source: Optimal Estimation and Generalization Bounds
- Keeping Score: Adaptive, Tuning-Free Loss Weighting for Score-Augmented Neural Ratio Estimation
- Tuning-Free Covariance-Aware Sequential Shrinkage for Multi-Source Estimation
- EntroPrefill: Renyi-Guided Context Pruning with Conditional Stability Guarantees for Retrieval-Augmented Generation
- The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation.
BibTeX
@misc{lad2026retrievalguided,
title = {{Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysis}},
author = {Bhargav Lad and Yifan Hao},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.14485},
eprint = {2609.14485},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.14485},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysis
AU - Bhargav Lad
AU - Yifan Hao
PY - 2026
DA - 2026/09/13
PB - arXiv
DO - 10.48550/arxiv.2609.14485
UR - https://arxiv.org/abs/2609.14485
AB - Retrieval-Guided Fine-Tuning (RAG-FT) incorporates retrieved data directly into the training objective, but the statistical consequences of noisy retrieval during training remain theoretically undercharacterized. We study this question by modeling RAG-FT as an estimation problem in a multi-task linear regression framework, using an OLS proxy for single-layer linear self-attention to obtain finite-sample risk bounds. Under homoscedastic retrieval noise, we show that retrieval failure decays exponentially with task separation relative to noise, and derive explicit finite-sample conditions under which RAG-FT achieves lower risk than both target-only and full-corpus training. We then introduce a Distance-Proportional Noise (DPN) model, in which retrieval quality degrades with rank, and compare two estimators under the same retrieval process: the OLS proxy and the literal, uniform-weight forward pass of linear self-attention. We prove that the attention estimator's bias diverges as Θ(n^2q) even under exact retrieval, while OLS risk remains Θ(d/n) for every noise exponent q>0. These results locate the instability not in noisy retrieval itself, but in the fixed, unweighted aggregation of the literal LSA forward pass, which reweighting by reliability empirically removes. We validate the predicted rate separation through direct simulation of the DPN model.
ER -