Experimentally Validated Digital Twins: Evaluating Workforce Training Programs in the Age of Big Data
Abstract
Modern program evaluation prioritizes evidence from randomized controlled trials (RCTs) because prior research found that observational estimates of program impacts often diverge substantially from experimental benchmarks. We revisit this finding --- which was based on small survey datasets --- in the era of large-scale administrative data. Using anonymized income tax records covering all American adults between 1999 and 2025, we estimate the effects of workforce training programs on earnings by using machine learning models to predict counterfactuals from rich data on prior work histories, demographics, and residential locations. Rather than testing whether our observational estimates are unbiased, we use 37 RCTs to estimate the distribution of their bias. We find that our machine learning estimates closely match the RCT estimates, with root mean squared error (RMSE) comparable to that of the RCTs themselves. Our method reduces RMSE by 60% relative to parametric estimators used in prior work because our larger samples and richer covariates permit more flexible, higher-dimensional models of the counterfactual. Applying our method to a broader set of sites and enrollment years beyond the experimental sample, we find that treatment effects on earnings vary substantially across locations and over time, even within a given program. Programs that place more participants into target-sector jobs generate the largest earnings gains, with impacts that persist even after participants leave their initial jobs. These findings provide guidance for improving job training programs and, more broadly, illustrate a scalable approach to program evaluation that uses observational data to construct experimentally validated ``digital twins.''
Related
- The Acculturation-Deculturation Dialectic at the Heart of ICT Teacher Training Programs
- CP-WSP: A Declarative CP-SAT Framework for Configurable Multi-Constraint Workforce Scheduling
- Geographic Disparities in Hospice Quality and Family Caregiver Experience: The Roles of Ownership, Social Vulnerability, and Workforce Capacity
- The optimality of an (s, S) hiring policy on a workforce planning problem with fixed recruitment costs and binomial turnover
- From Metaheuristics to Exact Methods: A CP-SAT Approach for Multi-Objective Healthcare Workforce Scheduling
BibTeX
@misc{chetty2026experimentally,
title = {{Experimentally Validated Digital Twins: Evaluating Workforce Training Programs in the Age of Big Data}},
author = {Raj Chetty and Jamie Fogel and Lawrence F. Katz and Kadeem Noray and Sonya Porter and James Reisinger},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.03938},
eprint = {2611.03938},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.03938},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Experimentally Validated Digital Twins: Evaluating Workforce Training Programs in the Age of Big Data
AU - Raj Chetty
AU - Jamie Fogel
AU - Lawrence F. Katz
AU - Kadeem Noray
AU - Sonya Porter
AU - James Reisinger
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.03938
UR - https://arxiv.org/abs/2611.03938
AB - Modern program evaluation prioritizes evidence from randomized controlled trials (RCTs) because prior research found that observational estimates of program impacts often diverge substantially from experimental benchmarks. We revisit this finding --- which was based on small survey datasets --- in the era of large-scale administrative data. Using anonymized income tax records covering all American adults between 1999 and 2025, we estimate the effects of workforce training programs on earnings by using machine learning models to predict counterfactuals from rich data on prior work histories, demographics, and residential locations. Rather than testing whether our observational estimates are unbiased, we use 37 RCTs to estimate the distribution of their bias. We find that our machine learning estimates closely match the RCT estimates, with root mean squared error (RMSE) comparable to that of the RCTs themselves. Our method reduces RMSE by 60% relative to parametric estimators used in prior work because our larger samples and richer covariates permit more flexible, higher-dimensional models of the counterfactual. Applying our method to a broader set of sites and enrollment years beyond the experimental sample, we find that treatment effects on earnings vary substantially across locations and over time, even within a given program. Programs that place more participants into target-sector jobs generate the largest earnings gains, with impacts that persist even after participants leave their initial jobs. These findings provide guidance for improving job training programs and, more broadly, illustrate a scalable approach to program evaluation that uses observational data to construct experimentally validated ``digital twins.''
ER -