Temporal-Difference Calibration of Black-Box Simulators
Abstract
We develop a derivative-free temporal-difference (TD) framework for calibrating black-box simulators using one-step transitions from a target system and noisy simulator-generated continuation values. The calibration parameter is defined through Bellman orthogonality, recovering the physical parameter under correct simulator specification and yielding a near-optimal L2 approximation of the policy value under misspecification. We propose three TD algorithms based on random-direction finite differences, distinguished by their query costs and reuse of noisy simulator evaluations. We establish almost-sure convergence, last-iterate mean-square error bounds, and central limit theorems for Polyak-Ruppert averages across four query-noise regimes. Our analysis reveals a specification-dependent bias phenomenon: for two algorithms, the finite-difference bias of the TD estimating equation vanishes at the calibration target under correct specification, even when the gradient surrogate remains biased. This permits a fixed perturbation radius and rate exponents approaching 1/2 in the simulator rollout budget, whereas misspecification generally recovers the classical Kiefer-Wolfowitz rates. We also develop a few-replicate online bootstrap that yields Student-t calibrated confidence intervals under negligible query noise, and identify a covariance mismatch that can invalidate conventional multiplier inference along Markov trajectories. Finally, a second-stage TD recursion corrects the remaining value discrepancy without additional simulator queries.
Related
- Online Inference in Distributional Temporal-Difference Learning
- On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces
- A Finite-Sample Analysis of Quantile Temporal-Difference Learning
- A Sharp Finite-Iteration Theory for Asynchronous Categorical Distributional Temporal-Difference Learning
- Spacetime Bartnik Mass Positivity and Temporal Monotonicity for Black Holes
BibTeX
@misc{zeng2026temporaldifference,
title = {{Temporal-Difference Calibration of Black-Box Simulators}},
author = {Min Zeng and Xin Wen and Xi Chen and Yichen Zhang and Wen-Xin Zhou},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.35311},
eprint = {2610.35311},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.35311},
}
RIS
TY - UNPB
TI - Temporal-Difference Calibration of Black-Box Simulators
AU - Min Zeng
AU - Xin Wen
AU - Xi Chen
AU - Yichen Zhang
AU - Wen-Xin Zhou
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2610.35311
UR - https://arxiv.org/abs/2610.35311
AB - We develop a derivative-free temporal-difference (TD) framework for calibrating black-box simulators using one-step transitions from a target system and noisy simulator-generated continuation values. The calibration parameter is defined through Bellman orthogonality, recovering the physical parameter under correct simulator specification and yielding a near-optimal L_2 approximation of the policy value under misspecification. We propose three TD algorithms based on random-direction finite differences, distinguished by their query costs and reuse of noisy simulator evaluations. We establish almost-sure convergence, last-iterate mean-square error bounds, and central limit theorems for Polyak-Ruppert averages across four query-noise regimes. Our analysis reveals a specification-dependent bias phenomenon: for two algorithms, the finite-difference bias of the TD estimating equation vanishes at the calibration target under correct specification, even when the gradient surrogate remains biased. This permits a fixed perturbation radius and rate exponents approaching 1/2 in the simulator rollout budget, whereas misspecification generally recovers the classical Kiefer-Wolfowitz rates. We also develop a few-replicate online bootstrap that yields Student-t calibrated confidence intervals under negligible query noise, and identify a covariance mismatch that can invalidate conventional multiplier inference along Markov trajectories. Finally, a second-stage TD recursion corrects the remaining value discrepancy without additional simulator queries.
ER -