Characterizing the Separability Gap in Prior-Data Fitted Networks
Abstract
Prior-data fitted networks (PFNs) are transformers pretrained on simulated datasets to approximate Bayesian posterior predictive distributions. Standard PFN theory assumes a separable task prior, under which the feature distribution and the mechanism generating labels from features are independent. We show that common PFN priors are non-separable, even in simple settings. Consequently, standard pretraining targets a predictive distribution based on a task posterior updated using both the labeled context and the query features. We characterize its difference from the predictive based on the context posterior alone and show that this difference vanishes under posterior concentration. Under misspecification (i.e., when the true joint distribution lies outside the prior's support), however, prior non-separability can favor better feature fit at the expense of conditional label fit. We illustrate the consequences for feature attribution, Bayesian optimization, and martingale checks of Bayesian behavior. The same dependence also enables PFNs to learn from additional unlabeled observations. We demonstrate this with SemiTabICL, a PFN explicitly pretrained to use these observations, which improves accuracy over a matched supervised baseline. Our findings highlight the need for explicit rules on unlabeled data use in PFN evaluation and establish the dependence between feature and label distributions as a consequential prior design choice.
Related
- Prior-data fitted networks for variable selection
- Prior-Data Fitted Networks for In-Context Off-Policy Evaluation
- How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?
- Anisotropic Trust Region Warping for Prior-Data Fitted Networks in Bayesian Optimisation
- Amortized Nonlinear Variable Selection with Prior-Data Fitted Networks
BibTeX
@misc{kobialka2026characterizing,
title = {{Characterizing the Separability Gap in Prior-Data Fitted Networks}},
author = {Julius Kobialka and Manuel Hülskamp and Thomas Nagler and David Rügamer},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.14989},
eprint = {2610.14989},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.14989},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Characterizing the Separability Gap in Prior-Data Fitted Networks
AU - Julius Kobialka
AU - Manuel Hülskamp
AU - Thomas Nagler
AU - David Rügamer
PY - 2026
DA - 2026/10/10
PB - arXiv
DO - 10.48550/arxiv.2610.14989
UR - https://arxiv.org/abs/2610.14989
AB - Prior-data fitted networks (PFNs) are transformers pretrained on simulated datasets to approximate Bayesian posterior predictive distributions. Standard PFN theory assumes a separable task prior, under which the feature distribution and the mechanism generating labels from features are independent. We show that common PFN priors are non-separable, even in simple settings. Consequently, standard pretraining targets a predictive distribution based on a task posterior updated using both the labeled context and the query features. We characterize its difference from the predictive based on the context posterior alone and show that this difference vanishes under posterior concentration. Under misspecification (i.e., when the true joint distribution lies outside the prior's support), however, prior non-separability can favor better feature fit at the expense of conditional label fit. We illustrate the consequences for feature attribution, Bayesian optimization, and martingale checks of Bayesian behavior. The same dependence also enables PFNs to learn from additional unlabeled observations. We demonstrate this with SemiTabICL, a PFN explicitly pretrained to use these observations, which improves accuracy over a matched supervised baseline. Our findings highlight the need for explicit rules on unlabeled data use in PFN evaluation and establish the dependence between feature and label distributions as a consequential prior design choice.
ER -