Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
Abstract
Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement update. This work introduces Regularized Dual Averaging Actor Critic (RDA2C), which assigns a distinct role to replay. In regularized dual averaging, the subsequent policy is determined by accumulated policy-improvement feedback, so historical advantage estimates contribute directly to the actor objective rather than solely to the most recent update. RDA2C stores state-action samples with critic-estimated advantage labels, fits a dual score model Zθ to the aggregated dataset, and derives the current policy from the accumulated score model using the entropy mirror map. In this way, replay defines an empirical dual objective from which the policy is computed. To analyze RDA2C, we establish a finite-time value-gap decomposition, separating the regularized dual-averaging term from errors due to stale-replay supervised fitting, critic bias, finite-buffer variance, and replay coverage, and stating the assumptions under which each error is bounded. RDA2C accepts advantage labels from any critic. With GAE labels, RDA2C outperforms PPO on six of eight MuJoCo tasks and eight of twelve Atari games. RDA2C also outperforms AAPDA, the closest dual-averaging baseline, on six of eight MuJoCo tasks. With twin-Q labels, RDA2C matches SAC at matched batch size and update frequency.
Related
- When to Follow the Teacher: Advantage-Based On-Policy Distillation
- Online Bayesian Model Averaging with Joint Uncertainty Quantification for Models and Regression Coefficients in Binary Regression
- A Robust Fast Greedy Model Averaging Method for High-Dimensional Regression with Outliers
- Does Policy Mirror Descent Converge in Policy with Non-Decomposable Mirror Maps? A Counterexample
- Notes on Uniform Policy Score Bounds and Lipschitz Continuity for Truncated Gaussian, Product Cauchy, Boltzmann, and Softmax Policies
BibTeX
@misc{peng2026policy,
title = {{Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression}},
author = {Nianli Peng and Geoffrey J. Gordon and Kianté Brantley},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.04638},
eprint = {2610.04638},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.04638},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
AU - Nianli Peng
AU - Geoffrey J. Gordon
AU - Kianté Brantley
PY - 2026
DA - 2026/10/03
PB - arXiv
DO - 10.48550/arxiv.2610.04638
UR - https://arxiv.org/abs/2610.04638
AB - Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement update. This work introduces Regularized Dual Averaging Actor Critic (RDA2C), which assigns a distinct role to replay. In regularized dual averaging, the subsequent policy is determined by accumulated policy-improvement feedback, so historical advantage estimates contribute directly to the actor objective rather than solely to the most recent update. RDA2C stores state-action samples with critic-estimated advantage labels, fits a dual score model Z_θ to the aggregated dataset, and derives the current policy from the accumulated score model using the entropy mirror map. In this way, replay defines an empirical dual objective from which the policy is computed. To analyze RDA2C, we establish a finite-time value-gap decomposition, separating the regularized dual-averaging term from errors due to stale-replay supervised fitting, critic bias, finite-buffer variance, and replay coverage, and stating the assumptions under which each error is bounded. RDA2C accepts advantage labels from any critic. With GAE labels, RDA2C outperforms PPO on six of eight MuJoCo tasks and eight of twelve Atari games. RDA2C also outperforms AAPDA, the closest dual-averaging baseline, on six of eight MuJoCo tasks. With twin-Q labels, RDA2C matches SAC at matched batch size and update frequency.
ER -