An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information
Abstract
Loss-difference conditional mutual information (ld-CMI) uses the smallest of the standard observations in the supersample hierarchy of generalization bounds: it measures what a learner's loss differences reveal about which candidate of each pair it was trained on. Accuracy is known to force information into the model; data processing does not carry such lower bounds to losses. We show, by bounding three moments of the loss differences, that accuracy also forces ld-CMI. For linear predictors with a smooth convex loss of nonzero slope at zero, such as the logistic loss, plus a regularizer whose curvature and growth are both of power r≥2, on product distributions over a scaled sign cube in dimension at least linear in n, every proper learner with expected excess risk at most ε on these distributions at the optimal sample size nasympε−2+2/r has worst-case ld-CMI of order n bits, and Θ(n/(1+(τ/ε)2)) bits under Gaussian noise of standard deviation τ on the loss differences. The same holds without a regularizer, at nasympε−2. Consequently, range-scaled ld-CMI bounds cannot vanish on these distributions, although every proper learner's generalization gap is O(n−1/2). We also show that model-level information does not determine noisy loss-difference information, and that the growth, slope and dimension conditions are needed, the last up to a logarithm.
Related
- A unifying theory for the evaluation of conditional Akaike information for mixed-effects models
- Conditional Fisher-Information Central Limit Theorems under Log-Concavity with Information-Theoretic Consequences
- Bias-Corrected Estimators for Joint Entropy, Conditional Entropy, and Mutual Information in the Discrete Bivariate Case: Theory and an Application to Motor Insurance
- Ex Ante and Scenario-Conditional Value of Information in Dynamic Portfolio Choice
- When Is Accuracy Evidence? A Unified Theory of Generalisation, Validation, and Information Fusion
BibTeX
@misc{yueksel2026accuracyinformation,
title = {{An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information}},
author = {Hazar Yueksel},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.09206},
eprint = {2610.09206},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.09206},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information
AU - Hazar Yueksel
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.09206
UR - https://arxiv.org/abs/2610.09206
AB - Loss-difference conditional mutual information (ld-CMI) uses the smallest of the standard observations in the supersample hierarchy of generalization bounds: it measures what a learner's loss differences reveal about which candidate of each pair it was trained on. Accuracy is known to force information into the model; data processing does not carry such lower bounds to losses. We show, by bounding three moments of the loss differences, that accuracy also forces ld-CMI. For linear predictors with a smooth convex loss of nonzero slope at zero, such as the logistic loss, plus a regularizer whose curvature and growth are both of power r≥2, on product distributions over a scaled sign cube in dimension at least linear in n, every proper learner with expected excess risk at most ε on these distributions at the optimal sample size nasympε^−2+2/r has worst-case ld-CMI of order n bits, and Θ(n/(1+(τ/ε)^2)) bits under Gaussian noise of standard deviation τ on the loss differences. The same holds without a regularizer, at nasympε^−2. Consequently, range-scaled ld-CMI bounds cannot vanish on these distributions, although every proper learner's generalization gap is O(n^−1/2). We also show that model-level information does not determine noisy loss-difference information, and that the growth, slope and dimension conditions are needed, the last up to a logarithm.
ER -