Feature Information Dynamics in Diffusion
Abstract
Diffusion models generate data through a continuum of denoising problems, and are widely observed to reveal coarse structure before fine detail. Yet, this intuition is mostly empirical and qualitative. We introduce feature information dynamics, an information-theoretic framework for localizing when a feature is generated during diffusion. Using the I-MMSE identity, we connect the rate of feature mutual information change to a gap between optimal unconditional and feature-conditional denoising losses, yielding practical estimators for feature information density. We further develop a chained decomposition that separates shared from incremental information in a feature hierarchy. We use this framework first to quantitatively confirm spectral autoregression in pixel diffusion, and then to extend the analysis beyond frequency: under a class → mask → Canny conditioning chain, the per-feature information densities differ across pixel, SDVAE, VAVAE, and RAE, exposing fundamental differences between these representations and suggesting that ordered generation could be beneficial for training diffusion models. Our code is available at https://github.com/AI4Science-WestlakeU/feature-information-dynamics.
Related
- Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
- Koopman Observers for Diffusion Acceleration: Correcting Feature Forecasts with Shallow Measurements
- Universal Distributed Feature Extraction Complementary to Side Information via H-Score
- Scalable, Likelihood-Free Calibration of Ice-Sheet Models with Deep Diffusion Emulators and Feature Matching
- DiBME-Net: Mirror-Boundary-Guided Diffusion-Band Feature Enhancement for RGB-D Mirror Segmentation
BibTeX
@misc{pan2026feature,
title = {{Feature Information Dynamics in Diffusion}},
author = {Jia-Shu Pan and Tao Zhang and Yufei Huang and Yanjun Sheng and Tailin Wu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.08626},
eprint = {2610.08626},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.08626},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Feature Information Dynamics in Diffusion
AU - Jia-Shu Pan
AU - Tao Zhang
AU - Yufei Huang
AU - Yanjun Sheng
AU - Tailin Wu
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.08626
UR - https://arxiv.org/abs/2610.08626
AB - Diffusion models generate data through a continuum of denoising problems, and are widely observed to reveal coarse structure before fine detail. Yet, this intuition is mostly empirical and qualitative. We introduce feature information dynamics, an information-theoretic framework for localizing when a feature is generated during diffusion. Using the I-MMSE identity, we connect the rate of feature mutual information change to a gap between optimal unconditional and feature-conditional denoising losses, yielding practical estimators for feature information density. We further develop a chained decomposition that separates shared from incremental information in a feature hierarchy. We use this framework first to quantitatively confirm spectral autoregression in pixel diffusion, and then to extend the analysis beyond frequency: under a class → mask → Canny conditioning chain, the per-feature information densities differ across pixel, SDVAE, VAVAE, and RAE, exposing fundamental differences between these representations and suggesting that ordered generation could be beneficial for training diffusion models. Our code is available at https://github.com/AI4Science-WestlakeU/feature-information-dynamics.
ER -