Rate-Distortion Theory of Gaussian Mixtures via Posterior Label Sampling
Abstract
We study the mean-squared error (MSE) rate-distortion function (RDF) of finite Gaussian mixture models (GMMs) in ℝd, for which neither the RDF nor an optimal test channel is known in closed form. It can be sandwiched between the genie-aided conditional RDF RX | T(D), which can be characterized in closed-form for a GMM, and RX | T(D)+I(X;T), where T denotes the latent component label. We show that the upper bound can be achieved operationally by the proposed Gibbs-relabelling method (GRM), which conveys a label from the posterior PT|X through \textit{channel simulation} at rate I(X;T), followed by vector quantization of the resulting exactly Gaussian conditional sources by reverse water-filling. Using the Shannon lower bound (SLB), we prove that GRM attains the SLB, and hence the RDF, of a general finite mixture if and only if every component attains its own SLB. For a GMM this holds throughout the fully active interval D/d<mink,mλk,m, where λk,m are the eigenvalues of the component covariances, and for scalar GMMs the condition is also necessary. Since GRM cannot operate below the label cost I(X;T), we propose a complementary stochastic scalar quantization (SSQ) scheme for low rates, which simulates an M-ary reconstruction index. We show that at low rates SSQ with M≥ 2 matches the Shannon RDF to first order: R(Dmax−δ)=δ/(2τ2)+o(δ), where τ2 is a directional sub-Gaussian proxy of the mixture, and quantify the price of a finite codebook through a strict rate gap at high rates and a distortion floor whose joint scaling in M and number of mixture components is characterized. Numerical results illustrate both regimes and the finite-blocklength cost of channel simulation.
Related
- Dirichlet Process Mixtures of Trees with Gaussian Process Splits: A Bayesian Nonparametric Framework with Posterior Contraction Rate
- Differential Privacy of Gaussian Process Posterior Sampling
- Rate-Distortion-Perception Tradeoff for Vector Gaussian Sources with Limited Common Randomness: Exact Characterization and Approximate Universality
- Pairwise Distortion Distribution with Graph-Based Importance Sampling for Quantization Error and Intrinsic Dimension Estimation
- A Self-Coupling Gaussian Extremal Inequality for Information-Constrained Optimal Transport with Applications to Rate-Distortion-Perception
BibTeX
@misc{wang2026ratedistortion,
title = {{Rate-Distortion Theory of Gaussian Mixtures via Posterior Label Sampling}},
author = {Weichen Wang and Deniz Gündüz},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.15549},
eprint = {2610.15549},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.15549},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Rate-Distortion Theory of Gaussian Mixtures via Posterior Label Sampling
AU - Weichen Wang
AU - Deniz Gündüz
PY - 2026
DA - 2026/10/11
PB - arXiv
DO - 10.48550/arxiv.2610.15549
UR - https://arxiv.org/abs/2610.15549
AB - We study the mean-squared error (MSE) rate-distortion function (RDF) of finite Gaussian mixture models (GMMs) in ℝ^d, for which neither the RDF nor an optimal test channel is known in closed form. It can be sandwiched between the genie-aided conditional RDF R_X | T(D), which can be characterized in closed-form for a GMM, and R_X | T(D)+I(X;T), where T denotes the latent component label. We show that the upper bound can be achieved operationally by the proposed Gibbs-relabelling method (GRM), which conveys a label from the posterior P_T|X through \textit{channel simulation} at rate I(X;T), followed by vector quantization of the resulting exactly Gaussian conditional sources by reverse water-filling. Using the Shannon lower bound (SLB), we prove that GRM attains the SLB, and hence the RDF, of a general finite mixture if and only if every component attains its own SLB. For a GMM this holds throughout the fully active interval D/d<min_k,mλ_k,m, where λ_k,m are the eigenvalues of the component covariances, and for scalar GMMs the condition is also necessary. Since GRM cannot operate below the label cost I(X;T), we propose a complementary stochastic scalar quantization (SSQ) scheme for low rates, which simulates an M-ary reconstruction index. We show that at low rates SSQ with M≥ 2 matches the Shannon RDF to first order: R(D_max−δ)=δ/(2τ^2)+o(δ), where τ^2 is a directional sub-Gaussian proxy of the mixture, and quantify the price of a finite codebook through a strict rate gap at high rates and a distortion floor whose joint scaling in M and number of mixture components is characterized. Numerical results illustrate both regimes and the finite-blocklength cost of channel simulation.
ER -