Comparing Corrupted Constrained Learning Problems
Abstract
A key result in statistics is the data processing inequality, originally proved by Blackwell (1951) and later refined by DeGroot (1962) in terms of statistical uncertainty. The latter statement claims that the Bayes risk achieved on raw data always undercuts the Bayes risk on a processed version of the same data. This seemingly contradicts empirical findings in machine learning: pre-training, representation learning, feature learning, data augmentation are techniques used to improve performance of machine learning models. We reconcile both worlds by simply accepting that machine learning problems are constrained learning problems: the model class used does not include all measurable functions. We present counterexamples showing that the classical data processing inequality fails to hold in such a setting. Hence, we formulate a generalized data processing inequality, requiring the constrained Bayes risk of a joint distribution (with respect to a loss function and a constrained model class) to lower bound the constrained Bayes risk on the stochastically modified data distribution, regardless of the choice of distribution. We show this inequality to be equivalent to a set containment condition on a specific function set induced by the loss and model class, called the superprediction set. Finally, we exploit our characterization, derive sufficient conditions for this containment and quantify the inequality-gap.
Related
- Solvability of a New Class of ψ-Fractional Unilateral Constrained Friction Problems
- A Safeguarded Projected-Gradient Framework for Complementarity Constrained Least Squares Problems
- Robust preconditioners for finite element approximations of control-state constrained optimal control problems governed by elliptic equations
- Online Feedback Optimization for Constrained Stochastic Problems with Decision-Dependent Distributions
- Pontryagin maximum principle for non-smooth state-constrained control problems over Wasserstein spaces
BibTeX
@misc{iacovissi2026comparing,
title = {{Comparing Corrupted Constrained Learning Problems}},
author = {Laura Iacovissi and Rabanus Derr and Robert C. Williamson},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2608.25745},
eprint = {2608.25745},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.25745},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Comparing Corrupted Constrained Learning Problems
AU - Laura Iacovissi
AU - Rabanus Derr
AU - Robert C. Williamson
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2608.25745
UR - https://arxiv.org/abs/2608.25745
AB - A key result in statistics is the data processing inequality, originally proved by Blackwell (1951) and later refined by DeGroot (1962) in terms of statistical uncertainty. The latter statement claims that the Bayes risk achieved on raw data always undercuts the Bayes risk on a processed version of the same data. This seemingly contradicts empirical findings in machine learning: pre-training, representation learning, feature learning, data augmentation are techniques used to improve performance of machine learning models. We reconcile both worlds by simply accepting that machine learning problems are constrained learning problems: the model class used does not include all measurable functions. We present counterexamples showing that the classical data processing inequality fails to hold in such a setting. Hence, we formulate a generalized data processing inequality, requiring the constrained Bayes risk of a joint distribution (with respect to a loss function and a constrained model class) to lower bound the constrained Bayes risk on the stochastically modified data distribution, regardless of the choice of distribution. We show this inequality to be equivalent to a set containment condition on a specific function set induced by the loss and model class, called the superprediction set. Finally, we exploit our characterization, derive sufficient conditions for this containment and quantify the inequality-gap.
ER -