An analysis of Mirror-Descent Soft Actor-Critic
Abstract
Soft Actor-Critic (SAC) is widely used for entropy-regularised reinforcement learning with continuous action spaces, and practical implementations perform only a few actor steps towards an evolving target. In this work, we prove convergence guarantees when the target policy arises from policy mirror descent and compare it with the classical Gibbs target. We derive sufficient conditions for the strong convexity and smoothness of the actor objective, characterised by the curvature of the Q-function estimate through the Legendre differential operator, and establish an 𝒪 (N−1/5) best-iterate finite-time convergence rate up to actor and critic approximation errors. Moreover, the mirror-descent step size λ directly controls the target drift and hence actor tracking error, whereas the analogous Gibbs bound contains a non-vanishing tracking term.
Related
- Explicit Coordinated Signal Timing Optimization Using Soft Actor–Critic and Model Predictive Control Along an Arterial
- Generative Actor-Critic with Soft Bridge Policies
- Viscosity-Informed Generative Actor-Critic for High-Dimensional Stochastic Optimal Control
- Neural Actor--Critic Learning with Minimax Critics and Reflected Langevin Sampling
- Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis
BibTeX
@misc{zorba2026analysis,
title = {{An analysis of Mirror-Descent Soft Actor-Critic}},
author = {Denis Zorba and Michal Valko},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.35466},
eprint = {2609.35466},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.35466},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - An analysis of Mirror-Descent Soft Actor-Critic
AU - Denis Zorba
AU - Michal Valko
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2609.35466
UR - https://arxiv.org/abs/2609.35466
AB - Soft Actor-Critic (SAC) is widely used for entropy-regularised reinforcement learning with continuous action spaces, and practical implementations perform only a few actor steps towards an evolving target. In this work, we prove convergence guarantees when the target policy arises from policy mirror descent and compare it with the classical Gibbs target. We derive sufficient conditions for the strong convexity and smoothness of the actor objective, characterised by the curvature of the Q-function estimate through the Legendre differential operator, and establish an 𝒪 (N^−1/5) best-iterate finite-time convergence rate up to actor and critic approximation errors. Moreover, the mirror-descent step size λ directly controls the target drift and hence actor tracking error, whereas the analogous Gibbs bound contains a non-vanishing tracking term.
ER -