σTransfer: Uncertainty Transfer from Small to Large Networks under μP
Abstract
Reliable predictive uncertainty in Laplace approximations depends critically on the prior precision, yet selecting it requires a posterior sweep that is prohibitively expensive for neural networks with billions of parameters. Under the Maximal Update Parametrization (μP), we derive a rescaling of the prior covariance that makes the selected precision stable as model width grows. This leads to σTransfer: we select the precision on a smaller model and zero-shot transfer it to the much larger model, i.e., without searching for the precision on the larger model at all. We show convergence of the prior kernel, posterior covariance, selected precision, and posterior-derived decisions under explicit conditions, and verify σTransfer across regression, image classification, and Transformer readouts. For example, measured precision-sweep speedups reach ∼ 5000× when transferring from width 128 to 4096 on MNIST, at a target-NLL degradation of $0.002$; transferring from a public 1B to 7B model gives a median search speedup of ∼ 2.3× (up to ∼ 330×), with a mean measured target-NLL increase below 10−4 across ten tasks. The same posterior stability also enables transfer of acquisition, OOD-detection, and abstention decisions without constructing a target posterior.
Related
- Small and large quasi-kernels: a weighted transfer theorem
- Guided Uncertainty-Aware Robust Domain Transfer
- Thermodynamic Concentration Inequalities: Controlling Uncertainty in Finite-Time and Small-Sample Thermodynamic Inference
- Safe Position and Attitude Control for Landing on Small Celestial Bodies with Gravitational Uncertainty
- Adaptive Uncertainty Quantification on Simplicial Complexes with Conformalized Simplicial Neural Networks
BibTeX
@misc{bergna2026σtransfer,
title = {{$σ$Transfer: Uncertainty Transfer from Small to Large Networks under $μ\mathrmP$}},
author = {Richard Bergna and Fernando Ruiz Mazo and Nicolò Felicioni and José Miguel Hernández-Lobato and Kamil Ciosek},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.11668},
eprint = {2610.11668},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.11668},
}
RIS
TY - UNPB
TI - σTransfer: Uncertainty Transfer from Small to Large Networks under μP
AU - Richard Bergna
AU - Fernando Ruiz Mazo
AU - Nicolò Felicioni
AU - José Miguel Hernández-Lobato
AU - Kamil Ciosek
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.11668
UR - https://arxiv.org/abs/2610.11668
AB - Reliable predictive uncertainty in Laplace approximations depends critically on the prior precision, yet selecting it requires a posterior sweep that is prohibitively expensive for neural networks with billions of parameters. Under the Maximal Update Parametrization (μP), we derive a rescaling of the prior covariance that makes the selected precision stable as model width grows. This leads to σTransfer: we select the precision on a smaller model and zero-shot transfer it to the much larger model, i.e., without searching for the precision on the larger model at all. We show convergence of the prior kernel, posterior covariance, selected precision, and posterior-derived decisions under explicit conditions, and verify σTransfer across regression, image classification, and Transformer readouts. For example, measured precision-sweep speedups reach ∼ 5000× when transferring from width 128 to 4096 on MNIST, at a target-NLL degradation of $0.002$; transferring from a public 1B to 7B model gives a median search speedup of ∼ 2.3× (up to ∼ 330×), with a mean measured target-NLL increase below 10^−4 across ten tasks. The same posterior stability also enables transfer of acquisition, OOD-detection, and abstention decisions without constructing a target posterior.
ER -