Controlling Dependence in Implicit Generative Models via Spread Mutual Information
Abstract
Mutual information (MI) provides an objective for suppressing or encouraging statistical dependence in implicit generative models. However, direct MI evaluation is challenging in implicit models due to typically intractable densities. A remedy is estimating the generator gradient from the difference between conditional and marginal scores. This score difference can, in turn, be estimated by differentiating a log density ratio learned through classification. This construction nevertheless faces two difficulties: (i) singular distributions need not admit the required score functions, and (ii) poor overlap can hinder density-ratio estimation. We therefore introduce Spread Mutual Information (SMI), a weighted integral of MI across noise levels obtained by applying a common spreading kernel to the generated variable. Gaussian spreading yields smooth, strictly positive conditional and marginal densities, extending the gradient construction to distributions that may originally be singular. Across a variaty of experiments, SMI consistently achieves effective dependence control among MI-based methods and remains competitive with established task-specific approaches.
Related
- LyapuFlow: Controlling Generative Flows with Lyapunov Feedback for Inverse Problems
- Shrinkage prior for implicit generative models
- Embedding Information in Policy Randomness: Distribution-Preserving Implicit Communication in MDPs
- DOMIC: Provably Calibrated Detection of Dependence-Structure Change Points via Density-Operator Mutual Information
- The Implicit Combinatorial Bijection in Gauss's Article 148 of the Disquisitiones Arithmeticae
BibTeX
@misc{yu2026controlling,
title = {{Controlling Dependence in Implicit Generative Models via Spread Mutual Information}},
author = {Jiahao Yu and Song Liu and José Miguel Hernández-Lobato and RuiKang OuYang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10021},
eprint = {2610.10021},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10021},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Controlling Dependence in Implicit Generative Models via Spread Mutual Information
AU - Jiahao Yu
AU - Song Liu
AU - José Miguel Hernández-Lobato
AU - RuiKang OuYang
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10021
UR - https://arxiv.org/abs/2610.10021
AB - Mutual information (MI) provides an objective for suppressing or encouraging statistical dependence in implicit generative models. However, direct MI evaluation is challenging in implicit models due to typically intractable densities. A remedy is estimating the generator gradient from the difference between conditional and marginal scores. This score difference can, in turn, be estimated by differentiating a log density ratio learned through classification. This construction nevertheless faces two difficulties: (i) singular distributions need not admit the required score functions, and (ii) poor overlap can hinder density-ratio estimation. We therefore introduce Spread Mutual Information (SMI), a weighted integral of MI across noise levels obtained by applying a common spreading kernel to the generated variable. Gaussian spreading yields smooth, strictly positive conditional and marginal densities, extending the gradient construction to distributions that may originally be singular. Across a variaty of experiments, SMI consistently achieves effective dependence control among MI-based methods and remains competitive with established task-specific approaches.
ER -