Divergence based Ranked Preference Alignment for LLMs
Abstract
Alignment of large language models typically relies on pairwise preference feedback, though ranked feedback can capture more comprehensive information across multiple responses within a single annotation. However, ranked-preference alignment is underexplored in practice compared to simpler pairwise approaches, potentially resulting in suboptimal behavior. Concurrently, recent research has conceptualized large language model alignment objectives as divergence estimators between aligned (preferred) and unaligned (less-preferred) response distributions, resulting in a family of objectives with an interpretable theoretical foundation and strong empirical performance. This work unifies these perspectives by introducing f-RPO, a f-divergence objective for ranked preference optimization that extends divergence-based alignment to ranked feedback and recovers pairwise preference alignment as a special case. Our theoretical analysis establishes rank-order consistency, showing that f-RPO preserves the underlying preference ordering at convergence. Empirical results indicate that total-variation and reverse-KL variants of the proposed objective consistently outperform ranked-alignment baselines in both ranking and generation quality across Qwen, Llama, & Gemma models ranging from 0.5B to 8B parameters.
Related
- FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
- Anchor-proofness in Voting with Ranked Ballots
- Close Races and Similarity Bias: A Bayesian Analysis of the Ranked Ballots for the Ballon d'Or
- Ranked Right, Sized Wrong: Ensemble Spread Fails Silently at Streamflow Extremes
- Single-Temperature Ranked Greedy Cluster Exchange for Max-Cut on Two-Dimensional Tori
BibTeX
@misc{haldar2026divergence,
title = {{Divergence based Ranked Preference Alignment for LLMs}},
author = {Rajdeep Haldar and Yue Xing and Qifan Song},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.25588},
eprint = {2610.25588},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.25588},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Divergence based Ranked Preference Alignment for LLMs
AU - Rajdeep Haldar
AU - Yue Xing
AU - Qifan Song
PY - 2026
DA - 2026/10/18
PB - arXiv
DO - 10.48550/arxiv.2610.25588
UR - https://arxiv.org/abs/2610.25588
AB - Alignment of large language models typically relies on pairwise preference feedback, though ranked feedback can capture more comprehensive information across multiple responses within a single annotation. However, ranked-preference alignment is underexplored in practice compared to simpler pairwise approaches, potentially resulting in suboptimal behavior. Concurrently, recent research has conceptualized large language model alignment objectives as divergence estimators between aligned (preferred) and unaligned (less-preferred) response distributions, resulting in a family of objectives with an interpretable theoretical foundation and strong empirical performance. This work unifies these perspectives by introducing f-RPO, a f-divergence objective for ranked preference optimization that extends divergence-based alignment to ranked feedback and recovers pairwise preference alignment as a special case. Our theoretical analysis establishes rank-order consistency, showing that f-RPO preserves the underlying preference ordering at convergence. Empirical results indicate that total-variation and reverse-KL variants of the proposed objective consistently outperform ranked-alignment baselines in both ranking and generation quality across Qwen, Llama, & Gemma models ranging from 0.5B to 8B parameters.
ER -