High-Value Training Data from Noisy, Diverse Sources
Abstract
Training state-of-the-art models increasingly requires learning from large, heterogeneous datasets whose examples vary in quality, domain relevance, and source of supervision. Dataweighting---assigning different weights to training examples---offers a general way to control how such data influences training, but learning effective weights at scale remains challenging. We develop a scalable approach to learning dataweights: given a small, high-quality target dataset, we use metagradients through the training process to efficiently optimize training-example weights directly for performance on the target domain. To scale beyond a fixed dataset and model, we learn parametric weighting rules that generalize to new and streaming examples and transfer from smaller models to substantially larger ones. We study reward modeling as a prototypical application, where preference data may be noisy, adversarial, synthetic, or mismatched with the target domain. Learned dataweights improve held-out performance, automatically adapt reward models to specialized domains, and allow mixtures of human and synthetic supervision to outperform either source alone.
Related
- On the critical value of void percolation in high dimensions
- The Capacity of the Noisy Torn Paper Channel and the Noisy Shuffling Channel
- Universal classical and quantum fluctuations in the large deviations of current of noisy quantum systems: Quantum symmetric simple exclusion and inclusion processes
- A weak Hellinger inequality for noisy Boolean channels
- Unital qubit channels are $3$-noisy
BibTeX
@misc{driss2026highvalue,
title = {{High-Value Training Data from Noisy, Diverse Sources}},
author = {Hedi Driss and Evan Frick and Anastasios N. Angelopoulos and Cho-Jui Hsieh and Yejin Huh and Tijana Zrnic and Andrew Ilyas},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.30421},
eprint = {2610.30421},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.30421},
}
RIS
TY - UNPB
TI - High-Value Training Data from Noisy, Diverse Sources
AU - Hedi Driss
AU - Evan Frick
AU - Anastasios N. Angelopoulos
AU - Cho-Jui Hsieh
AU - Yejin Huh
AU - Tijana Zrnic
AU - Andrew Ilyas
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.30421
UR - https://arxiv.org/abs/2610.30421
AB - Training state-of-the-art models increasingly requires learning from large, heterogeneous datasets whose examples vary in quality, domain relevance, and source of supervision. Dataweighting---assigning different weights to training examples---offers a general way to control how such data influences training, but learning effective weights at scale remains challenging. We develop a scalable approach to learning dataweights: given a small, high-quality target dataset, we use metagradients through the training process to efficiently optimize training-example weights directly for performance on the target domain. To scale beyond a fixed dataset and model, we learn parametric weighting rules that generalize to new and streaming examples and transfer from smaller models to substantially larger ones. We study reward modeling as a prototypical application, where preference data may be noisy, adversarial, synthetic, or mismatched with the target domain. Learned dataweights improve held-out performance, automatically adapt reward models to specialized domains, and allow mixtures of human and synthetic supervision to outperform either source alone.
ER -