Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control
Abstract
We develop a model-free policy gradient method for discrete-time mean-field control (MFC). In MFC, the policy affects the objective both through the controlled dynamics and through the population distribution. Standard REINFORCE estimators capture the first effect but not the second. We introduce Transport REINFORCE, a transport map-based approach that perturbs a suitable transformation of the population distribution to estimate this missing mean-field contribution. The method applies to both finite and continuous state spaces. In finite state spaces, we perturb the population distribution directly on the probability simplex through a convex combination of the current population weights and random weights. In continuous state spaces, we project the population distribution onto the manifold of Gaussian mixtures, and then randomize it via a transport map that ensures the perturbed law remains within this manifold. We prove consistency of the perturbed objective and gradient as the perturbation vanishes, and derive bias and mean-square error bounds for the resulting sample-based gradient estimator. Numerical experiments on several MFC benchmarks show that Transport REINFORCE improves over standard REINFORCE.
Related
- Policy Evaluation and Policy Gradient for Continuous-Time Markov Regime-Switching Systems
- (Cheap) Stochastic Policy Gradient Converges with High Probability for Linear Quadratic Regulator
- Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradient
- Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
- Optimality of Threshold Policies for Heat Storage Optimization and a Policy Gradient Algorithm
BibTeX
@misc{jamal2026randomized,
title = {{Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control}},
author = {Adonis Jamal and Samy Mekkaoui and Yadh Hafsi and Huyên Pham},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.11619},
eprint = {2610.11619},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.11619},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control
AU - Adonis Jamal
AU - Samy Mekkaoui
AU - Yadh Hafsi
AU - Huyên Pham
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.11619
UR - https://arxiv.org/abs/2610.11619
AB - We develop a model-free policy gradient method for discrete-time mean-field control (MFC). In MFC, the policy affects the objective both through the controlled dynamics and through the population distribution. Standard REINFORCE estimators capture the first effect but not the second. We introduce Transport REINFORCE, a transport map-based approach that perturbs a suitable transformation of the population distribution to estimate this missing mean-field contribution. The method applies to both finite and continuous state spaces. In finite state spaces, we perturb the population distribution directly on the probability simplex through a convex combination of the current population weights and random weights. In continuous state spaces, we project the population distribution onto the manifold of Gaussian mixtures, and then randomize it via a transport map that ensures the perturbed law remains within this manifold. We prove consistency of the perturbed objective and gradient as the perturbation vanishes, and derive bias and mean-square error bounds for the resulting sample-based gradient estimator. Numerical experiments on several MFC benchmarks show that Transport REINFORCE improves over standard REINFORCE.
ER -