Generative Actor-Critic with Soft Bridge Policies

Abstract

Expressive generative policies such as diffusion and flow models are appealing for maximum-entropy (MaxEnt) online reinforcement learning because of their ability to model multimodal and highly non-Gaussian action distributions. However, training effective soft generative policies faces two obstacles that often arise together. First, marginal action densities are often unavailable, so existing methods typically rely on entropy bounds, heuristic proxies or approximations. Second, iterative shared-parameter samplers raise inference cost and require backpropagation through time over repeated network evaluations, increasing memory cost and potentially destabilizing policy optimization. These obstacles motivate us to seek a generative actor-critic method that exposes a tractable soft objective while generating each action in just one network evaluation. To this end, we propose soft generative actor-critic (SoftGAC), whose actor is a single stochastic feedforward network with internal Gaussian bridge layers. This explicit bridge structure allows us to lift the soft objective to path space, yielding an analytical path-KL regularizer against a high-entropy reference process for both actor improvement and the soft Bellman bootstrap, without estimating the endpoint entropy. Crucially, the actor generates each action in only one network evaluation under a total parameter budget comparable to strong baselines, substantially reducing inference cost relative to the existing iterative diffusion and flow policies. It also avoids long unrolled backpropagation through time. Across challenging continuous-control benchmarks, our method achieves competitive or superior returns compared with strong diffusion and flow policies, demonstrating a clear advantage in the compute-return tradeoff.

An analysis of Mirror-Descent Soft Actor-Critic
Denis Zorba, Michal Valko
arXiv28 Sep 2026PreprintCC BY 4.0Full textOptimisation & control StatisticsarXiv
Viscosity-Informed Generative Actor-Critic for High-Dimensional Stochastic Optimal Control
Alen E. Golpashin, Gokul Puthumanaillam, Melkior Ornik +1
arXiv18 Sep 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Explicit Coordinated Signal Timing Optimization Using Soft Actor–Critic and Model Predictive Control Along an Arterial
Kun Zhang, Helai Huang, Hanchu Zhou +3
Mathematics5 Nov 2026Peer-reviewedCC BY 4.0General mathematicsCrossref
Neural Actor--Critic Learning with Minimax Critics and Reflected Langevin Sampling
Kunqi Huang, Lican Kang, Hongrui Shu +1
arXiv2 Nov 2026PreprintarXiv licenceFull textStatisticsarXiv
Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis
Yuling Jiao, Lican Kang, Jerry Zhijian Yang +1
arXiv27 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv

Cite

BibTeX RIS

BibTeX

@misc{he2026generative,
  title = {{Generative Actor-Critic with Soft Bridge Policies}},
  author = {Ke He and Le He and Shunpu Tang and Yafei Wang and Lisheng Fan},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2605.08733},
  eprint = {2605.08733},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.08733},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Generative Actor-Critic with Soft Bridge Policies
AU  - Ke He
AU  - Le He
AU  - Shunpu Tang
AU  - Yafei Wang
AU  - Lisheng Fan
PY  - 2026
DA  - 2026/10/14
PB  - arXiv
DO  - 10.48550/arxiv.2605.08733
UR  - https://arxiv.org/abs/2605.08733
AB  - Expressive generative policies such as diffusion and flow models are appealing for maximum-entropy (MaxEnt) online reinforcement learning because of their ability to model multimodal and highly non-Gaussian action distributions. However, training effective soft generative policies faces two obstacles that often arise together. First, marginal action densities are often unavailable, so existing methods typically rely on entropy bounds, heuristic proxies or approximations. Second, iterative shared-parameter samplers raise inference cost and require backpropagation through time over repeated network evaluations, increasing memory cost and potentially destabilizing policy optimization. These obstacles motivate us to seek a generative actor-critic method that exposes a tractable soft objective while generating each action in just one network evaluation. To this end, we propose soft generative actor-critic (SoftGAC), whose actor is a single stochastic feedforward network with internal Gaussian bridge layers. This explicit bridge structure allows us to lift the soft objective to path space, yielding an analytical path-KL regularizer against a high-entropy reference process for both actor improvement and the soft Bellman bootstrap, without estimating the endpoint entropy. Crucially, the actor generates each action in only one network evaluation under a total parameter budget comparable to strong baselines, substantially reducing inference cost relative to the existing iterative diffusion and flow policies. It also avoids long unrolled backpropagation through time. Across challenging continuous-control benchmarks, our method achieves competitive or superior returns compared with strong diffusion and flow policies, demonstrating a clear advantage in the compute-return tradeoff.
ER  -

Abstract

Related

Cite