ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control
Abstract
Language model post-training is often bottlenecked by the need for human-collected preference data, which is expensive and difficult to scale. Reinforcement learning from AI feedback (RLAIF) style approaches that leverage pseudo labels offer an abundant alternative but introduce systematic biases that degrade downstream alignment. Recent general-purpose semi-supervised methods correct for teacher bias using a small set of human-labeled examples, but suffer from high variance especially when human annotations are scarce. To this end, we propose ABC-Align, leveraging abundant pseudo label signal to minimize variance and applying a lightweight, adaptive correction grounded in the human-labeled subset. The correction strength is tuned automatically during training using plug-in estimates of the relevant bias--variance quantities. On LLM alignment with RLHF, DPO, and GRPO where human feedback is scarce, we empirically demonstrate that ABC-Align achieves superior performance over prior semi-supervised baselines in a series of experiments on an increasing scale. Our code is available at https://github.com/SewoongLab/abc-align .
Related
- PPI++: Efficient prediction-powered inference
- Beyond Point Predictions: Distribution-Informed Prediction-Powered Inference
- Covariance-Calibrated Prediction-Powered Inference
- Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation
- Prediction-Powered Data Fusion for Treatment Effect Estimation
BibTeX
@misc{frankel2026abcalign,
title = {{ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control}},
author = {Eric Frankel and Banghua Zhu and Sewoong Oh and Lillian J. Ratliff},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.34374},
eprint = {2609.34374},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.34374},
}
RIS
TY - UNPB
TI - ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control
AU - Eric Frankel
AU - Banghua Zhu
AU - Sewoong Oh
AU - Lillian J. Ratliff
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2609.34374
UR - https://arxiv.org/abs/2609.34374
AB - Language model post-training is often bottlenecked by the need for human-collected preference data, which is expensive and difficult to scale. Reinforcement learning from AI feedback (RLAIF) style approaches that leverage pseudo labels offer an abundant alternative but introduce systematic biases that degrade downstream alignment. Recent general-purpose semi-supervised methods correct for teacher bias using a small set of human-labeled examples, but suffer from high variance especially when human annotations are scarce. To this end, we propose ABC-Align, leveraging abundant pseudo label signal to minimize variance and applying a lightweight, adaptive correction grounded in the human-labeled subset. The correction strength is tuned automatically during training using plug-in estimates of the relevant bias--variance quantities. On LLM alignment with RLHF, DPO, and GRPO where human feedback is scarce, we empirically demonstrate that ABC-Align achieves superior performance over prior semi-supervised baselines in a series of experiments on an increasing scale. Our code is available at https://github.com/SewoongLab/abc-align .
ER -