PACE: Plug-and-Play Contextual Embedding for Feature Screening with Pretrained Tabular Foundation Models
Abstract
In high-dimensional tabular learning, feature screening provides a lightweight, model-agnostic way to remove irrelevant features before model fitting. However, scoring raw values directly can miss nonlinear or distributional structure. We introduce PACE (Plug-and-Play Contextual Embedding), which inserts a frozen tabular foundation model (TFM) column encoder before an existing feature-scoring rule, expanding each feature into a higher-dimensional contextual representation. Across controlled studies, PACE improves raw-space screening of complex nonlinear dependence with only modest additional encoding cost. These gains translate to downstream prediction on TALENT datasets: PACE-DC improves binary AUC by 0.077 and multiclass macro-AUC by 0.064, with a median normalized RMSE improvement of 0.063 across ten learners. Matched random-weight and random-feature controls show that PACE gains from pretrained structure beyond generic dimensional expansion. PACE further achieves favorable performance--time trade-offs against task-fitted selectors and attribution-based methods, positioning pretrained column geometry as a reusable upstream primitive for tabular learning.
Related
- Policy Regret for Embedding Model Routing: Contextual Bandits with Low-Rank Experts
- On the Duality between Feature and Sample Screening
- Multicollinearity-agnostic feature screening for non-Euclidean responses: a factor adjusted approach
- Model-free and Distributionally Robust Feature Screening with False Discovery Control for High-Dimensional Heterogeneous Data
- Residual-Based Causal Discovery via Pretrained In-Context Quantile Regression
BibTeX
@misc{qin2026pace,
title = {{PACE: Plug-and-Play Contextual Embedding for Feature Screening with Pretrained Tabular Foundation Models}},
author = {Qi Qin and Erbo Li and Ting Wei and Zizhou Huang and Zixuan Qin and Wu Wang and Yifan Sun},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.23574},
eprint = {2609.23574},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.23574},
}
RIS
TY - UNPB
TI - PACE: Plug-and-Play Contextual Embedding for Feature Screening with Pretrained Tabular Foundation Models
AU - Qi Qin
AU - Erbo Li
AU - Ting Wei
AU - Zizhou Huang
AU - Zixuan Qin
AU - Wu Wang
AU - Yifan Sun
PY - 2026
DA - 2026/09/20
PB - arXiv
DO - 10.48550/arxiv.2609.23574
UR - https://arxiv.org/abs/2609.23574
AB - In high-dimensional tabular learning, feature screening provides a lightweight, model-agnostic way to remove irrelevant features before model fitting. However, scoring raw values directly can miss nonlinear or distributional structure. We introduce PACE (Plug-and-Play Contextual Embedding), which inserts a frozen tabular foundation model (TFM) column encoder before an existing feature-scoring rule, expanding each feature into a higher-dimensional contextual representation. Across controlled studies, PACE improves raw-space screening of complex nonlinear dependence with only modest additional encoding cost. These gains translate to downstream prediction on TALENT datasets: PACE-DC improves binary AUC by 0.077 and multiclass macro-AUC by 0.064, with a median normalized RMSE improvement of 0.063 across ten learners. Matched random-weight and random-feature controls show that PACE gains from pretrained structure beyond generic dimensional expansion. PACE further achieves favorable performance--time trade-offs against task-fitted selectors and attribution-based methods, positioning pretrained column geometry as a reusable upstream primitive for tabular learning.
ER -