Causal DAG-Guided Reinforcement Learning for Long-Term Physical Activity Maintenance: Using Prior Data to Inform Future Deployment
Abstract
Digital health interventions can provide scalable, personalized support for physical activity (PA), yet sustaining PA over the long-term remains difficult. Because affective experiences with PA are important for activity sustainment, we developed an online reinforcement learning (RL) system for the forthcoming ADAPT factorial trial that personalizes twice-daily activity suggestions aimed at improving affective associations with PA. This setting poses several challenges: participant-specific data are scarce relative to the candidate state space, important states and rewards are partially observed, interventions are frequent while the primary outcome is measured weekly, and activity suggestions can both improve PA-related experiences and increase intervention burden. We address these challenges using an expert-specified causal directed acyclic graph as an integrated RL design tool, guiding state construction, estimation of partially observed states, delayed credit assignment, and representation of beneficial and adverse intervention pathways. We further construct a longitudinal simulator calibrated from a prior micro-randomized trial to compare candidate RL designs under realistic state evolution, missingness, and participant heterogeneity before deployment. Together, this work illustrates how causal domain knowledge and prior trial data can make long-horizon, participant-specific RL more practical for digital health interventions targeting sustained behavior change.
Related
- Offline Approximate Value Iteration: A Reinforcement Learning Approach for Satellite Constellation Deployment
- A Cyber Physical Framework for Intelligent Fault Detection and Predictive Maintenance in Digital Power Grids
- Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
- Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems
- Class-Conditioned Diffusion Augmentation and Explainable In-Context Ensemble Learning for Imbalanced Maintenance Action Classification in Aircraft Systems
BibTeX
@misc{liu2026causal,
title = {{Causal DAG-Guided Reinforcement Learning for Long-Term Physical Activity Maintenance: Using Prior Data to Inform Future Deployment}},
author = {Xueqing Liu and Young Hyun Cho and Bhanu Teja Gullapalli and Predrag Klasnja and Susan Murphy},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.36019},
eprint = {2610.36019},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.36019},
}
RIS
TY - UNPB
TI - Causal DAG-Guided Reinforcement Learning for Long-Term Physical Activity Maintenance: Using Prior Data to Inform Future Deployment
AU - Xueqing Liu
AU - Young Hyun Cho
AU - Bhanu Teja Gullapalli
AU - Predrag Klasnja
AU - Susan Murphy
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2610.36019
UR - https://arxiv.org/abs/2610.36019
AB - Digital health interventions can provide scalable, personalized support for physical activity (PA), yet sustaining PA over the long-term remains difficult. Because affective experiences with PA are important for activity sustainment, we developed an online reinforcement learning (RL) system for the forthcoming ADAPT factorial trial that personalizes twice-daily activity suggestions aimed at improving affective associations with PA. This setting poses several challenges: participant-specific data are scarce relative to the candidate state space, important states and rewards are partially observed, interventions are frequent while the primary outcome is measured weekly, and activity suggestions can both improve PA-related experiences and increase intervention burden. We address these challenges using an expert-specified causal directed acyclic graph as an integrated RL design tool, guiding state construction, estimation of partially observed states, delayed credit assignment, and representation of beneficial and adverse intervention pathways. We further construct a longitudinal simulator calibrated from a prior micro-randomized trial to compare candidate RL designs under realistic state evolution, missingness, and participant heterogeneity before deployment. Together, this work illustrates how causal domain knowledge and prior trial data can make long-horizon, participant-specific RL more practical for digital health interventions targeting sustained behavior change.
ER -