How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?
Abstract
Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in \textit{rare} settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.
Related
- Debiased Prior-Data Fitted Networks for Amortized Bayesian Semi-Parametric Inference
- Anisotropic Trust Region Warping for Prior-Data Fitted Networks in Bayesian Optimisation
- SelectPFN: Amortized Bayesian Variable Selection for Sparse Nonlinear Regression with Prior-Data Fitted Networks
- Prior-data fitted networks for variable selection
- Prior-Data Fitted Networks for In-Context Off-Policy Evaluation
BibTeX
@misc{hulskamp2026well,
title = {{How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?}},
author = {Manuel Hülskamp and Julius Kobialka and Emanuel Sommer and Thomas Nagler and David Rügamer},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.13764},
eprint = {2610.13764},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.13764},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?
AU - Manuel Hülskamp
AU - Julius Kobialka
AU - Emanuel Sommer
AU - Thomas Nagler
AU - David Rügamer
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2610.13764
UR - https://arxiv.org/abs/2610.13764
AB - Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in \textit{rare} settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.
ER -