Projected Primal–Dual Gradient Preference Optimization Under Category-Wise Split-Conformal Safety Constraints for Large Language Model Post-Training
- 1School of Information Science and Engineering, Dalian Polytechnic University, Ganjingzi District, Dalian 116034, China
- 2General Graduate School, Yonsei University, Seoul 03722, Republic of Korea
- 3School of Technology and Management, Kyung Hee University, Seoul 02447, Republic of Korea
Abstract
Preference optimisation improves helpfulness, but a single aggregate safety penalty lets frequent harm categories dominate the objective while rare ones bear the residual risk. This paper models post-training as a vector-constrained programme: the pairwise preference loss is minimised under one risk tolerance per harm category. The solver is a projected primal–dual gradient recursion that prices each category by its own residual violation. For a nonconvex objective we bound three horizon-averaged residuals—primal stationarity, dual regret and the average constraint value—for the unbiased, unclipped recursion and a uniformly drawn iterate; the implemented recursion is compared empirically with a no-EMA, no-clipping form of itself. Certification stays separate from optimisation: on an untouched calibration split, a multiplicity-adjusted fixed-sequence test gives release thresholds valid for all categories at once under correct category assignment, with an additive bound for learned routers. It certifies the scoring and filtering of corpus-distributed candidate responses, not text the trained policy generates. In one exploratory setting—one Qwen2.5-0.5B-Instruct backbone, PKU-SafeRLHF, three seeds—the recursion meets the per-category tolerance that direct preference optimisation and scalar-constrained schemes violate, cutting the worst-category miss rate at the training threshold from 0.176 to 0.057 at unchanged preference accuracy. The certified operating point barely moves: most of the constraint’s effect is a score translation shared by the harmful and benign responses of a prompt, which calibration provably absorbs. Whether a training constraint holds and whether the deployed gate is safer are different questions.
Related
- An Adaptive Projected-Gradient Algorithm for Sample-Average Approximations of Stochastic Multi-Objective Optimization
- Gradient-Free Methods for Stochastic Convex Optimization with Stochastic Functional Constraints
- Broximal Gradient Descent: A Projection-Free Sister of Projected Gradient Descent
- A Safeguarded Projected-Gradient Framework for Complementarity Constrained Least Squares Problems
- Projected Gradient Method on Hadamard Manifolds
BibTeX
@article{jiang2026projected,
title = {{Projected Primal–Dual Gradient Preference Optimization Under Category-Wise Split-Conformal Safety Constraints for Large Language Model Post-Training}},
author = {Haochen Jiang and Yulin Tu},
journal = {Mathematics},
year = {2026},
month = oct,
doi = {10.3390/math14193638},
url = {https://www.mdpi.com/2227-7390/14/19/3638},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Projected Primal–Dual Gradient Preference Optimization Under Category-Wise Split-Conformal Safety Constraints for Large Language Model Post-Training
AU - Haochen Jiang
AU - Yulin Tu
PY - 2026
DA - 2026/10/08
JO - Mathematics
DO - 10.3390/math14193638
UR - https://www.mdpi.com/2227-7390/14/19/3638
AB - Preference optimisation improves helpfulness, but a single aggregate safety penalty lets frequent harm categories dominate the objective while rare ones bear the residual risk. This paper models post-training as a vector-constrained programme: the pairwise preference loss is minimised under one risk tolerance per harm category. The solver is a projected primal–dual gradient recursion that prices each category by its own residual violation. For a nonconvex objective we bound three horizon-averaged residuals—primal stationarity, dual regret and the average constraint value—for the unbiased, unclipped recursion and a uniformly drawn iterate; the implemented recursion is compared empirically with a no-EMA, no-clipping form of itself. Certification stays separate from optimisation: on an untouched calibration split, a multiplicity-adjusted fixed-sequence test gives release thresholds valid for all categories at once under correct category assignment, with an additive bound for learned routers. It certifies the scoring and filtering of corpus-distributed candidate responses, not text the trained policy generates. In one exploratory setting—one Qwen2.5-0.5B-Instruct backbone, PKU-SafeRLHF, three seeds—the recursion meets the per-category tolerance that direct preference optimisation and scalar-constrained schemes violate, cutting the worst-category miss rate at the training threshold from 0.176 to 0.057 at unchanged preference accuracy. The certified operating point barely moves: most of the constraint’s effect is a score translation shared by the harmful and benign responses of a prompt, which calibration provably absorbs. Whether a training constraint holds and whether the deployed gate is safer are different questions.
SN - 2227-7390
ER -