Neural Actor--Critic Learning with Minimax Critics and Reflected Langevin Sampling
Abstract
Actor--critic methods rely on critic gradients to improve continuous-action policies, yet finite-sample guarantees must control both value and gradient errors and their propagation through the actor. We develop a neural actor-critic procedure that combines a regularized minimax Bellman estimator over sigmoid networks with a finite reflected Langevin sampler driven by the fitted critic gradient. Under uniform Bellman regularity and covered reset sampling, a positive high-order action-derivative penalty yields value and action-gradient risk bounds with classical nonparametric minimax exponents up to logarithmic factors. We propagate estimation, mixing, discretization, temperature, and policy-iteration errors to obtain return guarantees, with refinements for bounded multimodal potentials and strongly concave regimes. Empirical studies across controlled simulations, logged continuous-control data, and a simulated decision task show favorable performance relative to the considered baselines.
Related
- Minimax-Optimality of Posterior Sampling for Reinforcement Learning
- Reflected Anchored Langevin Algorithms
- Reflected Proximal Dynamics for Accelerated High-Accuracy Log-Concave Sampling
- Wasserstein Bounds for Euler and Randomized Midpoint Langevin Sampling with Noisy Gradients
- Sharp dimensional analysis of midpoint methods for Langevin sampling
BibTeX
@misc{huang2026neural,
title = {{Neural Actor--Critic Learning with Minimax Critics and Reflected Langevin Sampling}},
author = {Kunqi Huang and Lican Kang and Hongrui Shu and Jincheng Ying},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.03352},
eprint = {2611.03352},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.03352},
}
RIS
TY - UNPB
TI - Neural Actor--Critic Learning with Minimax Critics and Reflected Langevin Sampling
AU - Kunqi Huang
AU - Lican Kang
AU - Hongrui Shu
AU - Jincheng Ying
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.03352
UR - https://arxiv.org/abs/2611.03352
AB - Actor--critic methods rely on critic gradients to improve continuous-action policies, yet finite-sample guarantees must control both value and gradient errors and their propagation through the actor. We develop a neural actor-critic procedure that combines a regularized minimax Bellman estimator over sigmoid networks with a finite reflected Langevin sampler driven by the fitted critic gradient. Under uniform Bellman regularity and covered reset sampling, a positive high-order action-derivative penalty yields value and action-gradient risk bounds with classical nonparametric minimax exponents up to logarithmic factors. We propagate estimation, mixing, discretization, temperature, and policy-iteration errors to obtain return guarantees, with refinements for bounded multimodal potentials and strongly concave regimes. Empirical studies across controlled simulations, logged continuous-control data, and a simulated decision task show favorable performance relative to the considered baselines.
ER -