FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
Abstract
Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics.
Related
- Divergence based Ranked Preference Alignment for LLMs
- Rotation groups, mediangle graphs, and periagroups: A unified point of view on Coxeter groups and graph products of groups
- Local Factors in the BSD Conjecture: A Unified Statistical View
- On Inherent Privacy of Posterior Sampling: A Unified Rényi-Divergence Framework
- Mapping for Approximation: A Unified View of Rescaled, Variably Scaled and Rational Kernel Methods
BibTeX
@misc{han2026flowcpo,
title = {{FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models}},
author = {Yansen Han and Shengyi Liao and Peng Sun and Deyuan Liu and Yuanxing Zhang and Pengfei Wan and Tao Lin},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.09905},
eprint = {2609.09905},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.09905},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
AU - Yansen Han
AU - Shengyi Liao
AU - Peng Sun
AU - Deyuan Liu
AU - Yuanxing Zhang
AU - Pengfei Wan
AU - Tao Lin
PY - 2026
DA - 2026/09/09
PB - arXiv
DO - 10.48550/arxiv.2609.09905
UR - https://arxiv.org/abs/2609.09905
AB - Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics.
ER -