Deformed Q-learning for High-Dimensional Dynamic Treatment Regimes
Abstract
In high-dimensional dynamic treatment regimes, nonlinear terminal outcome models complicate upstream conditional-mean modeling and inference based on recursively estimated continuation values. We introduce a Deformed Q-learning procedure for general action spaces that accommodates a terminal-stage GLM, random designs, conditional heteroscedasticity, and fine-grained action discretization. The procedure changes the recursive criterion by propagating the terminal deformed optimal Q-function through sparse block-wise linear conditional-mean models. Under the stated models and regularity conditions, we establish consistent and asymptotically normal estimators, accounting for downstream estimation uncertainty in inference for optimal Q-values, deformed optimal Q-values, and their contrasts. We bound population regret relative to the original terminal mean-outcome objective without explicit stage-wise accumulation. Estimation and inference also extend to a growing number of stages under conditions uniform across stages. Simulation studies demonstrate accurate estimation and variable selection. In an empirical application to sepsis management using the MIMIC-III database, the proposed framework supports stage-specific value inference, while an auxiliary action-alignment analysis provides an exploratory signal that early-stage concordance with the learned recommendation may be associated with improved survival.
Related
- Estimating Optimal Dynamic Treatment Regimes for Personalized Education: A Tutorial and Applications with Machine Learning
- Nonparametric Efficient Estimation of Dynamic Treatment Regimes with Competing Risks
- Sharp Causal Bounds for Dynamic Treatment Regimes
- Functional dynamic mode decomposition: Learning infinite-dimensional systems from data
- Augmented Patient Preference Incorporated Reinforcement Learning (APP-RL) to Estimate the Optimal Dynamic Treatment Regime
BibTeX
@misc{cheng2026deformed,
title = {{Deformed Q-learning for High-Dimensional Dynamic Treatment Regimes}},
author = {Ming Cheng and Zhao Chen and Christina Dan Wang},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.00929},
eprint = {2611.00929},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.00929},
}
RIS
TY - UNPB
TI - Deformed Q-learning for High-Dimensional Dynamic Treatment Regimes
AU - Ming Cheng
AU - Zhao Chen
AU - Christina Dan Wang
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.00929
UR - https://arxiv.org/abs/2611.00929
AB - In high-dimensional dynamic treatment regimes, nonlinear terminal outcome models complicate upstream conditional-mean modeling and inference based on recursively estimated continuation values. We introduce a Deformed Q-learning procedure for general action spaces that accommodates a terminal-stage GLM, random designs, conditional heteroscedasticity, and fine-grained action discretization. The procedure changes the recursive criterion by propagating the terminal deformed optimal Q-function through sparse block-wise linear conditional-mean models. Under the stated models and regularity conditions, we establish consistent and asymptotically normal estimators, accounting for downstream estimation uncertainty in inference for optimal Q-values, deformed optimal Q-values, and their contrasts. We bound population regret relative to the original terminal mean-outcome objective without explicit stage-wise accumulation. Estimation and inference also extend to a growing number of stages under conditions uniform across stages. Simulation studies demonstrate accurate estimation and variable selection. In an empirical application to sepsis management using the MIMIC-III database, the proposed framework supports stage-specific value inference, while an auxiliary action-alignment analysis provides an exploratory signal that early-stage concordance with the learned recommendation may be associated with improved survival.
ER -