Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization

Abstract

Training a machine learning (ML) model is an optimization process that typically requires a set of examples with ground-truth labels beforehand. Obtaining these ground truths usually requires laborious human effort and can be costly. The scalability of human labeling becomes a concern when there are a large number of data points that need to be labeled. To address this practical issue, we consider an optimization setting where a subset of samples has ground-truth labels, while the remaining data points were labeled by an ML predictor, which could potentially assign incorrect labels to those points. We propose both first-order and zeroth-order stochastic optimization methods for training on datasets consisting of a mix of true-labeled and ML-labeled samples. We further conduct comprehensive empirical studies from training a linear regression model to fine-tuning a large language model (LLM) to showcase the effectiveness of the proposed algorithms for training with ML-labeled data.

Accelerated Stochastic Zeroth-Order Quasar-Convex Optimization
Eméric Gbaguidi, Julien Hermant
HAL17 Sep 2026PreprintCC BY-NC-ND 4.0Full textOptimisation & controlHAL
Stochastic Lie-Bracket Approximations for Zeroth-Order Optimization on Manifolds
Mahmoud Abdelgalil, Miroslav Krstic, Jorge I. Poveda
arXiv29 Sep 2026PreprintCC BY 4.0Full textOptimisation & controlarXiv
Multilevel Monte Carlo for Zeroth-Order Stochastic Bilevel Optimization with Inexact Lower Level Solves
Neil K. Chada, Mohammad S. Salehi, Mathias Staudigl +1
arXiv10 Oct 2026PreprintCC BY-NC-SA 4.0Full textNumerical analysis & scientific computing Optimisation & controlarXiv
An Ω(κ_y⁸ε⁻⁶) Lower Bound for Stochastic NC-SC Bilevel Optimization with First-order Oracles
Zhihao Gu, Qilong Wu, Junchi Yang
arXiv18 Sep 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Optimal Stochastic Bilevel Optimization with First-Order Oracles
Linxuan Pan, Junchi Yang
arXiv1 Oct 2026PreprintCC BY 4.0Full textOptimisation & controlarXiv

Cite

BibTeX RIS

BibTeX

@misc{sfyraki2026training,
  title = {{Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization}},
  author = {Maria-Eleni Sfyraki and Jun-Kun Wang},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.26790},
  eprint = {2610.26790},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26790},
}

RIS

TY  - UNPB
TI  - Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization
AU  - Maria-Eleni Sfyraki
AU  - Jun-Kun Wang
PY  - 2026
DA  - 2026/10/19
PB  - arXiv
DO  - 10.48550/arxiv.2610.26790
UR  - https://arxiv.org/abs/2610.26790
AB  - Training a machine learning (ML) model is an optimization process that typically requires a set of examples with ground-truth labels beforehand. Obtaining these ground truths usually requires laborious human effort and can be costly. The scalability of human labeling becomes a concern when there are a large number of data points that need to be labeled. To address this practical issue, we consider an optimization setting where a subset of samples has ground-truth labels, while the remaining data points were labeled by an ML predictor, which could potentially assign incorrect labels to those points. We propose both first-order and zeroth-order stochastic optimization methods for training on datasets consisting of a mix of true-labeled and ML-labeled samples. We further conduct comprehensive empirical studies from training a linear regression model to fine-tuning a large language model (LLM) to showcase the effectiveness of the proposed algorithms for training with ML-labeled data.
ER  -

Abstract

Related

Cite