Prior Predictive Matching for Bayesian LoRA
Abstract
Bayesian variants of Low-Rank Adaptation (LoRA) provide principled uncertainty estimation when fine-tuning large foundation models, but require specifying the prior over the low-rank weight updates. We address this hyperparameter selection problem by introducing a closed-form prior-predictive estimator that operates before fine-tuning. We analyze a second-order expansion of the network and quantify how the low-rank prior drives function-space deviations from the pretrained outputs. We also show that these prior-predictive quantities depend on the prior only through the product of the adaptation rank and the fourth power of the prior scale. Because the training targets share a single draw of the latent weight update, we can leverage the covariance between example residuals to derive a moment-matching estimator. Furthermore, our theoretical analysis establishes that the optimal prior precision scales as r−1/2 with the rank r, allowing a single estimate to generalize across all ranks. Controlled experiments across regression and classification tasks validate our approximation and show that our estimator recovers the true prior scale. On RoBERTa-large, our approach improves predictive negative log-likelihood (NLL) and calibration over both MAP and marginal-likelihood selection without requiring validation data, while achieving competitive out-of-distribution detection.
Related
- How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?
- Learning Informative Prior with Infinite-Dimensional Continuous Normalizing Flow for Bayesian Inverse Problem
- Weak Factor Recovery in Sparse Bayesian Factor Models under Exchangeable Shrinkage Process Prior
- Anisotropic Trust Region Warping for Prior-Data Fitted Networks in Bayesian Optimisation
- bayprior: Structured Bayesian Prior Elicitation, Conflict Diagnostics, and Regulatory Reporting
BibTeX
@misc{kusmierczyk2026prior,
title = {{Prior Predictive Matching for Bayesian LoRA}},
author = {Tomasz Kuśmierczyk and Eliezer de Souza da Silva and Marcin Sendera},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.35457},
eprint = {2610.35457},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.35457},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Prior Predictive Matching for Bayesian LoRA
AU - Tomasz Kuśmierczyk
AU - Eliezer de Souza da Silva
AU - Marcin Sendera
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2610.35457
UR - https://arxiv.org/abs/2610.35457
AB - Bayesian variants of Low-Rank Adaptation (LoRA) provide principled uncertainty estimation when fine-tuning large foundation models, but require specifying the prior over the low-rank weight updates. We address this hyperparameter selection problem by introducing a closed-form prior-predictive estimator that operates before fine-tuning. We analyze a second-order expansion of the network and quantify how the low-rank prior drives function-space deviations from the pretrained outputs. We also show that these prior-predictive quantities depend on the prior only through the product of the adaptation rank and the fourth power of the prior scale. Because the training targets share a single draw of the latent weight update, we can leverage the covariance between example residuals to derive a moment-matching estimator. Furthermore, our theoretical analysis establishes that the optimal prior precision scales as r^−1/2 with the rank r, allowing a single estimate to generalize across all ranks. Controlled experiments across regression and classification tasks validate our approximation and show that our estimator recovers the true prior scale. On RoBERTa-large, our approach improves predictive negative log-likelihood (NLL) and calibration over both MAP and marginal-likelihood selection without requiring validation data, while achieving competitive out-of-distribution detection.
ER -