Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning
Abstract
Direct Preference Optimization (DPO) treats every reported comparison as correct, yet preference data contain reversed, tied or otherwise noisy comparisons. RW-DPO gives each comparison an unknown reliability weight wi∈[0,1] that enters the likelihood linearly -- DPO is w≡1, a reversed comparison wi=0 -- and estimates the weights jointly with the policy under a minimax-concave penalty on −log wi. The penalty keeps every comparison the model agrees with at exactly wi=1 and acts only on those it contradicts, through closed-form weight updates at negligible cost over DPO. We prove that recovering all flipped comparisons from a single annotation per comparison requires margins of order log n. Furthermore, in a linear-index model, starting from an initial fit pointing in the correct direction, our estimator jointly consistently recovers the preference direction and weights, as well as identifies the flipped comparisons above this margin scale, even when a fixed fraction of comparisons is flipped. In simulations RW-DPO has the lowest average estimation error among DPO, noise-rate corrections and recent robust methods, with far more precise flags than recent label-correction methods. In DPO fine-tuning of a language model with 10-30% flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method with corrections concentrated on the flipped labels.
Related
- Statistical Learning Theory for Direct Preference Optimization
- Linking COPD Prevalence with Income Distribution: A Spatial Heterogeneous Compositional Regression via Geographically Weighted Penalized Approach
- A Safeguarded Accelerated Proximal ADMM Algorithm for Solving Structured Cardinality Penalized Optimization Problems
- Restarted Fixed-Point Acceleration of Proximal ADMM for Capped-ℓ1 Penalized Optimization
- Robust Direct Preference Optimization under Preference Label-Flip Attacks
BibTeX
@misc{luo2026reliabilityweighted,
title = {{Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning}},
author = {Bin Luo and Param Venkat Vivek Kesireddy and Anil Vallepu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.41588},
eprint = {2610.41588},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.41588},
}
RIS
TY - UNPB
TI - Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning
AU - Bin Luo
AU - Param Venkat Vivek Kesireddy
AU - Anil Vallepu
PY - 2026
DA - 2026/10/30
PB - arXiv
DO - 10.48550/arxiv.2610.41588
UR - https://arxiv.org/abs/2610.41588
AB - Direct Preference Optimization (DPO) treats every reported comparison as correct, yet preference data contain reversed, tied or otherwise noisy comparisons. RW-DPO gives each comparison an unknown reliability weight w_i∈[0,1] that enters the likelihood linearly -- DPO is w≡1, a reversed comparison w_i=0 -- and estimates the weights jointly with the policy under a minimax-concave penalty on −log w_i. The penalty keeps every comparison the model agrees with at exactly w_i=1 and acts only on those it contradicts, through closed-form weight updates at negligible cost over DPO. We prove that recovering all flipped comparisons from a single annotation per comparison requires margins of order log n. Furthermore, in a linear-index model, starting from an initial fit pointing in the correct direction, our estimator jointly consistently recovers the preference direction and weights, as well as identifies the flipped comparisons above this margin scale, even when a fixed fraction of comparisons is flipped. In simulations RW-DPO has the lowest average estimation error among DPO, noise-rate corrections and recent robust methods, with far more precise flags than recent label-correction methods. In DPO fine-tuning of a language model with 10-30% flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method with corrections concentrated on the flipped labels.
ER -