Optimal Transport Meets Reinforcement Learning: A Survey
Abstract
Reinforcement learning (RL) algorithms frequently compare probability distributions, such as state visitation distributions induced by policies and experts, action distributions from learned policies and offline datasets, or transition distributions from learned models and environments. However, commonly used divergences may become ineffective when these distributions overlap weakly, which is frequently encountered in imitation learning, offline RL, and deployment under distribution shift. Optimal transport (OT) offers an alternative by measuring the cost of \emph{moving} probability mass from one distribution to another under a ground cost that encodes task geometry. This survey covers how OT is used inside RL objectives and algorithms. For each method, we identify: the role OT plays, the distributions compared, the OT formulation used, and the treatment of temporal structure. Beyond categorising existing methods, we discuss the motivations behind different OT choices, practical considerations such as cost design and computational challenges, and highlight open problems including scalable trajectory-level transport, principled handling of mass mismatch, and theoretical analysis for OT-regularised RL.
Related
- Brenier Meets Adversarial Training: Optimal Transport Geometry for Robust Learning
- Optimal Recovery Meets Bayesian Learning: Where Worst-Case Bounds Pay Off
- Optimal Value Inference for Reinforcement Learning
- Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Model
- Augmented Patient Preference Incorporated Reinforcement Learning (APP-RL) to Estimate the Optimal Dynamic Treatment Regime
BibTeX
@misc{zhu2026optimal,
title = {{Optimal Transport Meets Reinforcement Learning: A Survey}},
author = {Yujie Zhu and Charles A. Hepburn and Matthew Thorpe and Giovanni Montana},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.01413},
eprint = {2610.01413},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.01413},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Optimal Transport Meets Reinforcement Learning: A Survey
AU - Yujie Zhu
AU - Charles A. Hepburn
AU - Matthew Thorpe
AU - Giovanni Montana
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.01413
UR - https://arxiv.org/abs/2610.01413
AB - Reinforcement learning (RL) algorithms frequently compare probability distributions, such as state visitation distributions induced by policies and experts, action distributions from learned policies and offline datasets, or transition distributions from learned models and environments. However, commonly used divergences may become ineffective when these distributions overlap weakly, which is frequently encountered in imitation learning, offline RL, and deployment under distribution shift. Optimal transport (OT) offers an alternative by measuring the cost of \emph{moving} probability mass from one distribution to another under a ground cost that encodes task geometry. This survey covers how OT is used inside RL objectives and algorithms. For each method, we identify: the role OT plays, the distributions compared, the OT formulation used, and the treatment of temporal structure. Beyond categorising existing methods, we discuss the motivations behind different OT choices, practical considerations such as cost design and computational challenges, and highlight open problems including scalable trajectory-level transport, principled handling of mass mismatch, and theoretical analysis for OT-regularised RL.
ER -