Ask to Learn: Query-Based Collaborative Reinforcement Learning
Abstract
Offline reinforcement learning is fundamentally limited by the coverage of the available data, while collecting additional trajectories through online exploration may be costly, unsafe, or infeasible. We study whether this limitation can be overcome without directly interacting with the underlying environment. Motivated by practical settings such as collaborator-mediated simulation, where a collaborator can provide information complementary to the offline data, we introduce query-based collaborative reinforcement learning to study how to acquire this information efficiently. The learner starts with a fixed offline dataset and adaptively queries state-time pairs; the collaborator samples an action according to a fixed, possibly suboptimal policy, and the simulator returns the corresponding reward and next state. For finite-horizon tabular problems, we derive an instance-dependent information-theoretic lower bound on the additional queries required to identify an ε-optimal policy and develop a local quadratic approximation. Guided by this characterization, we develop an adaptive query-allocation algorithm whose query complexity nearly matches the local benchmark under suitable conditions. Experiments on multiple synthetic and semi-synthetic environments demonstrate the empirical efficiency of the proposed method.
Related
- Wait to Learn: Timing the Gradient Return in Over-the-Air Split Learning over an Aging Single-Antenna Channel
- Planning to Learn: Real-Time Robust Adaptive Control on the GPU with Guaranteed Future Learning
- Y-Learn: Design and Initial Implementation of an OBE-Based Learning Platform for Group Work Monitoring and Peer Review Assessment
- A Query Is Not a Commitment: Learning to Correct Expert Answers in Online Deferral
- What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control
BibTeX
@misc{zhang2026learn,
title = {{Ask to Learn: Query-Based Collaborative Reinforcement Learning}},
author = {Kaicheng Zhang and Hongqian Niu and Laixi Shi and Didong Li and Guannan Qu and Zhun Deng},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.27606},
eprint = {2610.27606},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.27606},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Ask to Learn: Query-Based Collaborative Reinforcement Learning
AU - Kaicheng Zhang
AU - Hongqian Niu
AU - Laixi Shi
AU - Didong Li
AU - Guannan Qu
AU - Zhun Deng
PY - 2026
DA - 2026/10/25
PB - arXiv
DO - 10.48550/arxiv.2610.27606
UR - https://arxiv.org/abs/2610.27606
AB - Offline reinforcement learning is fundamentally limited by the coverage of the available data, while collecting additional trajectories through online exploration may be costly, unsafe, or infeasible. We study whether this limitation can be overcome without directly interacting with the underlying environment. Motivated by practical settings such as collaborator-mediated simulation, where a collaborator can provide information complementary to the offline data, we introduce query-based collaborative reinforcement learning to study how to acquire this information efficiently. The learner starts with a fixed offline dataset and adaptively queries state-time pairs; the collaborator samples an action according to a fixed, possibly suboptimal policy, and the simulator returns the corresponding reward and next state. For finite-horizon tabular problems, we derive an instance-dependent information-theoretic lower bound on the additional queries required to identify an ε-optimal policy and develop a local quadratic approximation. Guided by this characterization, we develop an adaptive query-allocation algorithm whose query complexity nearly matches the local benchmark under suitable conditions. Experiments on multiple synthetic and semi-synthetic environments demonstrate the empirical efficiency of the proposed method.
ER -