Learning to Fluctuate: Statistical Foundations for Causal Tabular Pretraining
Abstract
Causal tabular foundation models amortize effect estimation across synthetic mechanisms, but latent-effect supervision rewards posterior shrinkage rather than encoding the repeated-sample response needed in a fixed deployment population. We introduce fluctuation-supervised pretraining (FSP): each synthetic table is labeled by its average treatment effect plus its efficient influence-function fluctuation; deployment remains a frozen forward pass. Along the path Tλ,P=θ(P)+λPnψP, we prove an endpoint transition: every fixed λ<1 retains label ambiguity of order (1−λ)2/n, whereas full fluctuation makes the Gaussian label observable and reduces optimal finite-stratum causal label-prediction risk to order n−2. A finite-pretraining bound combines label, network, episode-sampling, and optimization errors; its sampling defect controls fixed-mechanism bias, mean squared error, variance, Gaussian approximation, and, with variance-head accuracy, studentized coverage. Complementary lower bounds separate local n−1 ATE risk from the log N/M excess risk of generic finite-dictionary episode learning. Experiments trace the learned sampling response. Across 24 nonlinear continuous-covariate cells at trained context lengths, continuous-row FSP lowers checkpoint-mean macro RMSE by 7.0% versus S-learner and wins all 12 weak-overlap cells; validation-selected Summary FSP deploys 11.6× faster per table in our warm one-thread benchmark. Under effect shift, matched Raw FSP lowers mean-checkpoint RMSE by 54.2% and teacher defect by 99.0% versus latent-effect supervision, and RMSE by 10.2% versus the released CausalPFN-S checkpoint. Known-effect semisynthesis tests coverage; two randomized-study evaluations show that lower RMSE can coexist with residual attenuation.
Related
- Statistical Foundations of Reinforcement Learning: A Non-Asymptotic Perspective
- Can Tabular Foundation Models Amortize Statistical Inference?
- A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
- Operator–Spectral χ‑Entropy and Hyperbolic Vacuum Structuring: Statistical Foundations of χ‑Reality
- Statistical Foundations for a Google Play User-Review Sentiment Index: Signal Fusion, Shrinkage, Distributional Validation, and Dynamic Smoothing
BibTeX
@misc{zhang2026learning,
title = {{Learning to Fluctuate: Statistical Foundations for Causal Tabular Pretraining}},
author = {Zhiheng Zhang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.26290},
eprint = {2609.26290},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.26290},
}
RIS
TY - UNPB
TI - Learning to Fluctuate: Statistical Foundations for Causal Tabular Pretraining
AU - Zhiheng Zhang
PY - 2026
DA - 2026/09/24
PB - arXiv
DO - 10.48550/arxiv.2609.26290
UR - https://arxiv.org/abs/2609.26290
AB - Causal tabular foundation models amortize effect estimation across synthetic mechanisms, but latent-effect supervision rewards posterior shrinkage rather than encoding the repeated-sample response needed in a fixed deployment population. We introduce fluctuation-supervised pretraining (FSP): each synthetic table is labeled by its average treatment effect plus its efficient influence-function fluctuation; deployment remains a frozen forward pass. Along the path T_λ,P=θ(P)+λP_nψ_P, we prove an endpoint transition: every fixed λ<1 retains label ambiguity of order (1−λ)^2/n, whereas full fluctuation makes the Gaussian label observable and reduces optimal finite-stratum causal label-prediction risk to order n^−2. A finite-pretraining bound combines label, network, episode-sampling, and optimization errors; its sampling defect controls fixed-mechanism bias, mean squared error, variance, Gaussian approximation, and, with variance-head accuracy, studentized coverage. Complementary lower bounds separate local n^−1 ATE risk from the log N/M excess risk of generic finite-dictionary episode learning. Experiments trace the learned sampling response. Across 24 nonlinear continuous-covariate cells at trained context lengths, continuous-row FSP lowers checkpoint-mean macro RMSE by 7.0% versus S-learner and wins all 12 weak-overlap cells; validation-selected Summary FSP deploys 11.6× faster per table in our warm one-thread benchmark. Under effect shift, matched Raw FSP lowers mean-checkpoint RMSE by 54.2% and teacher defect by 99.0% versus latent-effect supervision, and RMSE by 10.2% versus the released CausalPFN-S checkpoint. Known-effect semisynthesis tests coverage; two randomized-study evaluations show that lower RMSE can coexist with residual attenuation.
ER -