LASER: Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RL
Abstract
While offline reinforcement learning (RL) enables policy optimization from static datasets without costly online interaction, it remains bottlenecked by the risk of executing out-of-distribution (OOD) actions. Recent approaches mitigate this by learning a behavior-cloning policy through flow matching and then performing RL within its constrained latent space. However, naively optimizing the latent policy can easily cause the policy to collapse into a brittle mode or exploit sharp artifacts of the learned critic. In this work, we find that entropy regularization is essential in latent-space RL for addressing these challenges. We introduce LASER, a novel offline RL algorithm that applies latent-space adjoint matching to achieve entropy-regularized latent-space RL with expressive flow policies while avoiding backpropagation through time. Through comprehensive experiments on 40 challenging OGBench tasks with varying dataset qualities, we show that LASER achieves state-of-the-art performance. Notably, LASER uses fixed method-specific hyperparameters across all tasks and outperforms the evaluated baselines, including those with task- and dataset-specific tuning, which highlights the robust applicability of LASER. Project website: https://mit-realm.github.io/laser/.
Related
- Weighted Adjoint Matching: Finetuning Flow Matching Models with Time-Dependent Control Cost
- Self-Consistent Adjoint Policy Iteration for Constrained Dynamic Portfolio Choice
- Regularized Coupling Maps on Matrix Polytopes: From Assignment to Graph Matching and Optimal Transport
- From Inverse Problems to Adjoint Gradients: A Discrete, Implementation-Oriented Tutorial on PDE-Constrained Optimization
- Scalable Pontryagin-Guided Adjoint-to-Control Recovery for Constrained Dynamic Portfolio Choice
BibTeX
@misc{zhang2026laser,
title = {{LASER: Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RL}},
author = {Songyuan Zhang and Oswin So and Eric Yang Yu and Matthew Cleaveland and Peter Crowley-Dolen and Chuchu Fan},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.08989},
eprint = {2610.08989},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.08989},
}
RIS
TY - UNPB
TI - LASER: Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RL
AU - Songyuan Zhang
AU - Oswin So
AU - Eric Yang Yu
AU - Matthew Cleaveland
AU - Peter Crowley-Dolen
AU - Chuchu Fan
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.08989
UR - https://arxiv.org/abs/2610.08989
AB - While offline reinforcement learning (RL) enables policy optimization from static datasets without costly online interaction, it remains bottlenecked by the risk of executing out-of-distribution (OOD) actions. Recent approaches mitigate this by learning a behavior-cloning policy through flow matching and then performing RL within its constrained latent space. However, naively optimizing the latent policy can easily cause the policy to collapse into a brittle mode or exploit sharp artifacts of the learned critic. In this work, we find that entropy regularization is essential in latent-space RL for addressing these challenges. We introduce LASER, a novel offline RL algorithm that applies latent-space adjoint matching to achieve entropy-regularized latent-space RL with expressive flow policies while avoiding backpropagation through time. Through comprehensive experiments on 40 challenging OGBench tasks with varying dataset qualities, we show that LASER achieves state-of-the-art performance. Notably, LASER uses fixed method-specific hyperparameters across all tasks and outperforms the evaluated baselines, including those with task- and dataset-specific tuning, which highlights the robust applicability of LASER. Project website: https://mit-realm.github.io/laser/.
ER -