CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Abstract
Feedback noise is pervasive in preference data, arising from subjective human judgments, annotation mistakes, and imperfect LLM judges. Standard reward modeling trained on such feedback yields a biased estimate of the clean risk. Although classical denoising methods are directly applicable to noisy feedback learning, they generally assume a uniform noise ratio across samples. However, this assumption fails for feedback data, where noise is intrinsically sample-wise. To address this challenge, we propose CleanRM, which uses a few clean feedback to calibrate sample-wise noise ratios via optimal transport. Specifically, CleanRM aligns the joint distribution of preferences reverted from noisy feedback with that of clean feedback and then trains the reward model using a noise-corrected surrogate loss. Theoretically, we prove that the resulting objective is unbiased with respect to the clean risk. Extensive experiments demonstrate that CleanRM outperforms state-of-the-art denoising baselines and yields more effective downstream RLHF alignment.
Related
- Iterative Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
- KKTCode: Asymptotically Optimal Linear Codes for Noisy Feedback AWGN Channels
- A physics-informed inverse modeling framework for Moose-Wolf dynamics from limited and noisy data in Isle Royale National Park
- Notes on Generative Modeling for Feedback Control and Planning
- A mechanistic modeling framework to interpret ACTH stimulation tests across HPA axis adaptation states and glucocorticoid feedback dynamics
BibTeX
@misc{pan2026cleanrm,
title = {{CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF}},
author = {Licheng Pan and Haocheng Yang and Yuzhou Chen and Yichen Sun and Yuan Lu and Yunfei Teng and Jiajun Fan and Guangyi Liu and Zhixuan Chu and Hao Wang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.21138},
eprint = {2610.21138},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.21138},
}
RIS
TY - UNPB
TI - CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
AU - Licheng Pan
AU - Haocheng Yang
AU - Yuzhou Chen
AU - Yichen Sun
AU - Yuan Lu
AU - Yunfei Teng
AU - Jiajun Fan
AU - Guangyi Liu
AU - Zhixuan Chu
AU - Hao Wang
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.21138
UR - https://arxiv.org/abs/2610.21138
AB - Feedback noise is pervasive in preference data, arising from subjective human judgments, annotation mistakes, and imperfect LLM judges. Standard reward modeling trained on such feedback yields a biased estimate of the clean risk. Although classical denoising methods are directly applicable to noisy feedback learning, they generally assume a uniform noise ratio across samples. However, this assumption fails for feedback data, where noise is intrinsically sample-wise. To address this challenge, we propose CleanRM, which uses a few clean feedback to calibrate sample-wise noise ratios via optimal transport. Specifically, CleanRM aligns the joint distribution of preferences reverted from noisy feedback with that of clean feedback and then trains the reward model using a noise-corrected surrogate loss. Theoretically, we prove that the resulting objective is unbiased with respect to the clean risk. Extensive experiments demonstrate that CleanRM outperforms state-of-the-art denoising baselines and yields more effective downstream RLHF alignment.
ER -