Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data
Abstract
Digital twins (DTs) and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems. Owing to the inevitable simulation-to-reality (sim-to-real) gap, however, augmentation may fail to improve the performance of the trained model on the real data distribution. This paper addresses the resulting decision problem: Given a real dataset, a candidate synthetic dataset, and a fixed learning algorithm, decide whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible. Two formulations are considered: a direct test on the mean loss difference between the two trained models, and a symmetry-based test on the paired loss difference, which trades a stronger null assumption for faster evidence accumulation. For the latter, we introduce the {adaptive e-process sign-flip test} (aeSFT), a doubly adaptive procedure that adapts both the number of Monte Carlo sign-flip rounds, and hence the computational cost, and the amount of real test data consumed. aeSFT yields anytime-valid Type-I error control, with no need to pre-specify the test-set size. Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing, matches the power of fixed-sample sign-flip testing and the paired t-test, while keeping the false-positive rate below the target level.
Related
- Synthetic data for open medical research: what practitioners should know
- A Synthetic Benchmark Dataset with Endogenous Marketing Spend for Validating Marketing Mix Models
- Reproducible benchmarking of four machine-learning classifiers on a large synthetic heart-disease competition dataset
- Q-MINO: A Minimal-Norm Method for Quantization-Aware Training
- A Unified Framework for Training-Dataset Aggregation and Exact Machine Unlearning via Gibbs Probability Measures
BibTeX
@misc{tao2026should,
title = {{Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data}},
author = {Zhenyu Tao and Wei Xu and Xiaohu You and Petar Popovski and Osvaldo Simeone},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.27996},
eprint = {2608.27996},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.27996},
}
RIS
TY - UNPB
TI - Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data
AU - Zhenyu Tao
AU - Wei Xu
AU - Xiaohu You
AU - Petar Popovski
AU - Osvaldo Simeone
PY - 2026
DA - 2026/08/28
PB - arXiv
DO - 10.48550/arxiv.2608.27996
UR - https://arxiv.org/abs/2608.27996
AB - Digital twins (DTs) and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems. Owing to the inevitable simulation-to-reality (sim-to-real) gap, however, augmentation may fail to improve the performance of the trained model on the real data distribution. This paper addresses the resulting decision problem: Given a real dataset, a candidate synthetic dataset, and a fixed learning algorithm, decide whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible. Two formulations are considered: a direct test on the mean loss difference between the two trained models, and a symmetry-based test on the paired loss difference, which trades a stronger null assumption for faster evidence accumulation. For the latter, we introduce the {adaptive e-process sign-flip test} (aeSFT), a doubly adaptive procedure that adapts both the number of Monte Carlo sign-flip rounds, and hence the computational cost, and the amount of real test data consumed. aeSFT yields anytime-valid Type-I error control, with no need to pre-specify the test-set size. Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing, matches the power of fixed-sample sign-flip testing and the paired t-test, while keeping the false-positive rate below the target level.
ER -