Model-assisted estimation with a training subsample: a two-phase sampling approach with design-based variance estimation
Abstract
When a flexible prediction model is fitted on a training subsample drawn from a probability sample, the model-assisted estimator actually reported arises from one realized partition, yet existing theory quantifies uncertainty only for partition-averaged, cross-fitted, or symmetrized versions of it. We represent the training subsample as a second phase of sampling and derive, exactly and for any algorithm, a two-term variance decomposition and the variance family linking the single-partition estimator to its Rao-Blackwellized average, whose design bias it shares. For cell-mean rules under equal-probability designs the estimator is, exactly, a convex combination of a poststratified plug-in at the training-sample size and the exactly unbiased single-split estimator of Sanguiao Sande and Zhang, which locates its bias precisely. For tree-type predictors the second-phase variance is computable in closed form when the cell structure is fixed or s-measurable, and in that regime, for cell counts H=o(n), its share of total variance grows with the number of cells -- a design-based mechanism distinct from residual shrinkage and consistent with the variance underestimation documented for flexible learners. We propose an analytic and a replication variance estimator, neither altering the point estimate, and evaluate them by simulation under simple random and stratified designs, in the adaptive regime and in an s-measurable configuration in which the theory is exact: in the populations studied, budgeting the second phase recovers much of the coverage lost by ignoring it, and the residual shortfall is attributable to the bias of the point estimator rather than to its variance.
Related
- Prescreen, Then Subsample: Efficient model training for large-scale biomedical data with correlated covariates
- Learning a Sampling-Free Variational DNN Plugin from Tiny Training Sets to Refine OOD Segmentation With Uncertainty Estimation
- Posterior Tempering Explains Variance Inflation in Linear and Generalized Linear Thompson Sampling
- Nearly-minimax variance estimation under rough random design
- Spherical-Design Oracles for Zeroth-Order Optimisation: Deterministic, Variance-Free Gradient Estimation
BibTeX
@misc{riano2026modelassisted,
title = {{Model-assisted estimation with a training subsample: a two-phase sampling approach with design-based variance estimation}},
author = {María Eugenia Riaño},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.04082},
eprint = {2609.04082},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.04082},
}
RIS
TY - UNPB
TI - Model-assisted estimation with a training subsample: a two-phase sampling approach with design-based variance estimation
AU - María Eugenia Riaño
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2609.04082
UR - https://arxiv.org/abs/2609.04082
AB - When a flexible prediction model is fitted on a training subsample drawn from a probability sample, the model-assisted estimator actually reported arises from one realized partition, yet existing theory quantifies uncertainty only for partition-averaged, cross-fitted, or symmetrized versions of it. We represent the training subsample as a second phase of sampling and derive, exactly and for any algorithm, a two-term variance decomposition and the variance family linking the single-partition estimator to its Rao-Blackwellized average, whose design bias it shares. For cell-mean rules under equal-probability designs the estimator is, exactly, a convex combination of a poststratified plug-in at the training-sample size and the exactly unbiased single-split estimator of Sanguiao Sande and Zhang, which locates its bias precisely. For tree-type predictors the second-phase variance is computable in closed form when the cell structure is fixed or s-measurable, and in that regime, for cell counts H=o(n), its share of total variance grows with the number of cells -- a design-based mechanism distinct from residual shrinkage and consistent with the variance underestimation documented for flexible learners. We propose an analytic and a replication variance estimator, neither altering the point estimate, and evaluate them by simulation under simple random and stratified designs, in the adaptive regime and in an s-measurable configuration in which the theory is exact: in the populations studied, budgeting the second phase recovers much of the coverage lost by ignoring it, and the residual shortfall is attributable to the bias of the point estimator rather than to its variance.
ER -