FERPO: Forward Entropy-Regularized Policy Optimization
Abstract
Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO).
Related
- Forward-Time Black-Scholes Reconstruction via Regularized Legendre Reduction
- Entropy-Regularized Optimal Transport for Time-Varying Multi-Agent Coverage Control
- Accelerating Sinkhorn for Entropy-Regularized Optimal Transport
- Fast Regularized Policy Mirror Descent with One-Step TD Updates
- Entropy-Regularized Finite-State Mean Field Control: Regularity and Stability
BibTeX
@misc{sanokowski2026ferpo,
title = {{FERPO: Forward Entropy-Regularized Policy Optimization}},
author = {Sebastian Sanokowski and Alireza Sarmadi and Majid Khadiv},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02198},
eprint = {2610.02198},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02198},
}
RIS
TY - UNPB
TI - FERPO: Forward Entropy-Regularized Policy Optimization
AU - Sebastian Sanokowski
AU - Alireza Sarmadi
AU - Majid Khadiv
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.02198
UR - https://arxiv.org/abs/2610.02198
AB - Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO).
ER -