Direct Knowledge Transfer for Offline Q-Learning
Abstract
Data-driven sequential decision making in marketing and healthcare can benefit from observations collected in related environments, particularly when data in the environment of interest are limited. Transferring such information requires accounting for differences in rewards and transition dynamics across environments. We study knowledge transfer for offline reinforcement learning through a joint estimation framework that directly integrates source information into target value-function learning. We propose Direct Transfer Fitted Q-Iteration (DFQI), which updates the target Q-function while simultaneously borrowing the source information through an L1 penalty that links the target coefficients to a weighted aggregate of source coefficients, with both determined jointly within each Bellman update. Using sieve approximation, we establish statistical guarantees for target estimation and policy learning in stationary discounted environments with common or heterogeneous transition kernels. Empirical studies further demonstrate the effectiveness of the proposed DFQI framework in transferring source information to improve policy learning with limited target data.
Related
- Offline Approximate Value Iteration: A Reinforcement Learning Approach for Satellite Constellation Deployment
- Faster Hamiltonian Monte Carlo by learning leapfrog scale: an offline randomized solution
- Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
- Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry
- Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents
BibTeX
@misc{luo2026direct,
title = {{Direct Knowledge Transfer for Offline Q-Learning}},
author = {Zhanye Luo and Guangyan Gan and Lin William Cong},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01942},
eprint = {2611.01942},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01942},
}
RIS
TY - UNPB
TI - Direct Knowledge Transfer for Offline Q-Learning
AU - Zhanye Luo
AU - Guangyan Gan
AU - Lin William Cong
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01942
UR - https://arxiv.org/abs/2611.01942
AB - Data-driven sequential decision making in marketing and healthcare can benefit from observations collected in related environments, particularly when data in the environment of interest are limited. Transferring such information requires accounting for differences in rewards and transition dynamics across environments. We study knowledge transfer for offline reinforcement learning through a joint estimation framework that directly integrates source information into target value-function learning. We propose Direct Transfer Fitted Q-Iteration (DFQI), which updates the target Q-function while simultaneously borrowing the source information through an L1 penalty that links the target coefficients to a weighted aggregate of source coefficients, with both determined jointly within each Bellman update. Using sieve approximation, we establish statistical guarantees for target estimation and policy learning in stationary discounted environments with common or heterogeneous transition kernels. Empirical studies further demonstrate the effectiveness of the proposed DFQI framework in transferring source information to improve policy learning with limited target data.
ER -