Marformer: A Transformer for Predicting Missing Data Distributions
Abstract
Real decisions are made under incomplete information. If we observe only some of the random variables we need, we can predict the others. The \textbf{conditional marginals} over the missing variables are the key ingredient for computing Bayes risk and Value of Information (VOI), the expected gain from acquiring one more observation before deciding. We present the Marformer, a Transformer trained to directly predict conditional marginals given any set of observed values. Like BERT, which is trained to predict missing words from context, the Marformer constructs a hidden-vector representation for each distribution p(Xi) and iteratively refines it through attention to other distributions p(Xj). Unlike generative approaches, the Marformer does not model the full joint distribution, requires no domain knowledge of the data-generating process, and makes all predictions in a single forward pass. We evaluate across three synthetic domains with missing data---Bayesian networks, discretized multivariate Gaussians, and structured annotation data. The Marformer can match or outperform classical missing-data methods, even when those methods are given the true model family and prior that generated the synthetic data. We also evaluate on a real annotation dataset, where the Marformer outperforms the evaluated baselines at the largest training size. In both cases, the Marformer is substantially faster than the evaluated generative baselines.
Related
- Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure
- Finding New Limit Points of Mahler Measure by Methods of Missing Data Restoration
- Weighted mean squares of ζ and L(·,χ4): closed forms, reciprocity, and a missing identity
- Functions determining few direction: the final missing case
- The missing two-point inequality in Weissler's conjecture
BibTeX
@misc{singh2026marformer,
title = {{Marformer: A Transformer for Predicting Missing Data Distributions}},
author = {Prabhav Singh and Xiheng Tom Wang and Haojun Shi and Jason Eisner},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.12379},
eprint = {2610.12379},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.12379},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Marformer: A Transformer for Predicting Missing Data Distributions
AU - Prabhav Singh
AU - Xiheng Tom Wang
AU - Haojun Shi
AU - Jason Eisner
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.12379
UR - https://arxiv.org/abs/2610.12379
AB - Real decisions are made under incomplete information. If we observe only some of the random variables we need, we can predict the others. The \textbf{conditional marginals} over the missing variables are the key ingredient for computing Bayes risk and Value of Information (VOI), the expected gain from acquiring one more observation before deciding. We present the Marformer, a Transformer trained to directly predict conditional marginals given any set of observed values. Like BERT, which is trained to predict missing words from context, the Marformer constructs a hidden-vector representation for each distribution p(X_i) and iteratively refines it through attention to other distributions p(X_j). Unlike generative approaches, the Marformer does not model the full joint distribution, requires no domain knowledge of the data-generating process, and makes all predictions in a single forward pass. We evaluate across three synthetic domains with missing data---Bayesian networks, discretized multivariate Gaussians, and structured annotation data. The Marformer can match or outperform classical missing-data methods, even when those methods are given the true model family and prior that generated the synthetic data. We also evaluate on a real annotation dataset, where the Marformer outperforms the evaluated baselines at the largest training size. In both cases, the Marformer is substantially faster than the evaluated generative baselines.
ER -