Analytical and Bootstrap Confidence Intervals for Double Machine Learning: Simulation Studies and an Application to Rural-Urban Differences in Obesity Prevalence
Abstract
Double Machine Learning (DML) is a popular approach for treatment effect estimation in various settings, which allows a wide range of flexible machine learning methods to be used for nuisance parameter estimation while preserving valid inference. In practice, however, applied researchers must choose among many machine learning algorithms for nuisance models, and the impact of this choice on the variance estimation of DML is not well characterized. We conduct a comprehensive simulation study to compare the coverage probability of DML confidence intervals across different machine learning algorithms. In this study, we compare (1) analytical confidence intervals derived by DML theory versus (2) bootstrap confidence interval. We use a set of learners including ordinary least squares, LASSO, Random Forest, LightGBM, and Neural Networks under different data generation settings. We evaluate the performance across difference settings by bias, confidence interval width, and most importantly, coverage probability. Our results show substantial variability in coverage performance across analytical and bootstrap confidence intervals, highlighting that learner choice plays a critical role in reliable DML inference. Surprisingly, we find that in many settings, when sample size increases, the coverage probability of both DML analytical and bootstrap confidence interval decreases. We further investigate coverage probabilities using a real dataset on rural urban differences among U.S. counties. The real data analysis discovers that (1) the model performance still varies by the learner choices and (2) greater rurality has a statistically significant increasing effect on county level obesity prevalence.
Related
- Active learning confidence measures for coupling strategies in digital twins integrating simulation and data-driven submodels
- Analytical properties of polygonal Stokes and BGG Hessian complexes with application to Kirchhoff--Love plates
- Bayesian Cluster-Adjusted Standardised Mean Differences: A Simulation Study
- Simulation and Application of the Transmuted Unit Burr XII Distribution in Real-World Data Analysis
- Mediation and moderation in distance-based regression with triangular fuzzy data: simulation evidence and an application to academic performance
BibTeX
@misc{xu2026analytical,
title = {{Analytical and Bootstrap Confidence Intervals for Double Machine Learning: Simulation Studies and an Application to Rural-Urban Differences in Obesity Prevalence}},
author = {Haozheng Xu and Siyuan Ma and Qingyan Xiang},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2607.29456},
eprint = {2607.29456},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.29456},
}
RIS
TY - UNPB
TI - Analytical and Bootstrap Confidence Intervals for Double Machine Learning: Simulation Studies and an Application to Rural-Urban Differences in Obesity Prevalence
AU - Haozheng Xu
AU - Siyuan Ma
AU - Qingyan Xiang
PY - 2026
DA - 2026/08/25
PB - arXiv
DO - 10.48550/arxiv.2607.29456
UR - https://arxiv.org/abs/2607.29456
AB - Double Machine Learning (DML) is a popular approach for treatment effect estimation in various settings, which allows a wide range of flexible machine learning methods to be used for nuisance parameter estimation while preserving valid inference. In practice, however, applied researchers must choose among many machine learning algorithms for nuisance models, and the impact of this choice on the variance estimation of DML is not well characterized. We conduct a comprehensive simulation study to compare the coverage probability of DML confidence intervals across different machine learning algorithms. In this study, we compare (1) analytical confidence intervals derived by DML theory versus (2) bootstrap confidence interval. We use a set of learners including ordinary least squares, LASSO, Random Forest, LightGBM, and Neural Networks under different data generation settings. We evaluate the performance across difference settings by bias, confidence interval width, and most importantly, coverage probability. Our results show substantial variability in coverage performance across analytical and bootstrap confidence intervals, highlighting that learner choice plays a critical role in reliable DML inference. Surprisingly, we find that in many settings, when sample size increases, the coverage probability of both DML analytical and bootstrap confidence interval decreases. We further investigate coverage probabilities using a real dataset on rural urban differences among U.S. counties. The real data analysis discovers that (1) the model performance still varies by the learner choices and (2) greater rurality has a statistically significant increasing effect on county level obesity prevalence.
ER -