Robust Direct Preference Optimization under Preference Label-Flip Attacks

Abstract

Direct Preference Optimization (DPO) is widely used for offline Reinforcement Learning from Human Feedback. However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy. To address this issue, we design robust DPO methods that can defend against such attacks. Building on the property that lable flips induce parameter-independent gradient shifts in log-linear DPO, we propose Trim-K Robust-DPO, which removes the K highest-impact pairs, and Soft-Trim-K Robust-DPO, which smoothly downweights them. We establish deterministic gradient perturbation bounds uniformly over all attack sets containing at most K flips, explicitly separating defense-induced bias from surviving attack effects. Under bounded-region assumptions, these bounds yield time-uniform parameter and policy deviation guarantees. We further quantify how quadratic regularization improves curvature while introducing additional bias. Experiments on Stanford Human Preferences, HH-RLHF, and UltraFeedback-Binarized validate our analysis.

Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning
Bin Luo, Param Venkat Vivek Kesireddy, Anil Vallepu
arXiv30 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Statistical Learning Theory for Direct Preference Optimization
Joel Wills, Yuanfan Li, Yiming Ying +1
arXiv19 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Simulated Annealing for Model‐Robust Partial Profile Choice Designs in Healthcare Preference Studies
Yicheng Mao, Roselinde Kessels
Statistics in medicine1 Oct 2026Peer-reviewedCC BYFull textStatisticsEurope PMC
A Simple Presentation of Robust Membership Inference Attacks
Renaud Gaucher, Ephraim Linder, Adam Smith +2
arXiv27 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Adaptive Preference Elicitation in Preference Robust CPT-Based Shortfall
Sainan Zhang, Erick Delage, Huifu Xu
arXiv1 Nov 2026PreprintarXiv licenceFull textOptimisation & controlarXiv

Cite

BibTeX RIS

BibTeX

@misc{yang2026robust,
  title = {{Robust Direct Preference Optimization under Preference Label-Flip Attacks}},
  author = {Chenye Yang and Weiyu Xu and Lifeng Lai},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.24945},
  eprint = {2610.24945},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.24945},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Robust Direct Preference Optimization under Preference Label-Flip Attacks
AU  - Chenye Yang
AU  - Weiyu Xu
AU  - Lifeng Lai
PY  - 2026
DA  - 2026/10/18
PB  - arXiv
DO  - 10.48550/arxiv.2610.24945
UR  - https://arxiv.org/abs/2610.24945
AB  - Direct Preference Optimization (DPO) is widely used for offline Reinforcement Learning from Human Feedback. However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy. To address this issue, we design robust DPO methods that can defend against such attacks. Building on the property that lable flips induce parameter-independent gradient shifts in log-linear DPO, we propose Trim-K Robust-DPO, which removes the K highest-impact pairs, and Soft-Trim-K Robust-DPO, which smoothly downweights them. We establish deterministic gradient perturbation bounds uniformly over all attack sets containing at most K flips, explicitly separating defense-induced bias from surviving attack effects. Under bounded-region assumptions, these bounds yield time-uniform parameter and policy deviation guarantees. We further quantify how quadratic regularization improves curvature while introducing additional bias. Experiments on Stanford Human Preferences, HH-RLHF, and UltraFeedback-Binarized validate our analysis.
ER  -

Abstract

Related

Cite