Information, Estimation, and Prediction for Discrete Memoryless Channels via Poissonization
Abstract
We develop information--estimation relations for an arbitrary discrete memoryless channel with finite input and output alphabets by varying the expected number of observations while keeping the channel fixed. A single hidden input is observed through a \textit{Poisson} number of conditionally independent channel uses, whose mean specifies the observation budget. The derivative of mutual information with respect to this budget equals the minimum expected relative-entropy loss for estimating the input's output distribution. The posterior average of the channel's output distributions achieves this minimum and is the Bayesian predictor for the next observation. When the input distribution, the channel, or both are misspecified, the relative entropy between the true and assumed observation laws is the integral of the excess prediction loss over the budget. We also extend these relations to finite-state processes taking values in the input alphabet and observed through the channel at independent Poisson sampling times. Both in the matched setting and under model mismatch, the expected prediction loss accumulated over time using past observations equals the corresponding loss using the full observation record (noncausal estimation), averaged over lower sampling rates. We also characterize how redundancy between two outputs selects among input distributions that achieve one-use capacity. For the coupon-collector channel, where each observation is sampled uniformly from an unknown subset, we derive the exact capacity loss of any input distribution. In particular, balanced subset families can substantially reduce the input alphabet while preserving the leading capacity terms.
Related
- Information Estimation with Discrete Diffusion
- On the discrete Fisher information
- Exact Common Information and Exact Channel Synthesis for Correlated Discrete Sources
- Parameter Estimation for Unnormalized Discrete Models via Empirically Localized Deformed Bregman Divergence
- Parameter estimation for graphon-interacting particle systems from discrete observations
BibTeX
@misc{gorle2026information,
title = {{Information, Estimation, and Prediction for Discrete Memoryless Channels via Poissonization}},
author = {Abhiram Gorle and Tsachy Weissman},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.27737},
eprint = {2610.27737},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.27737},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Information, Estimation, and Prediction for Discrete Memoryless Channels via Poissonization
AU - Abhiram Gorle
AU - Tsachy Weissman
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.27737
UR - https://arxiv.org/abs/2610.27737
AB - We develop information--estimation relations for an arbitrary discrete memoryless channel with finite input and output alphabets by varying the expected number of observations while keeping the channel fixed. A single hidden input is observed through a \textit{Poisson} number of conditionally independent channel uses, whose mean specifies the observation budget. The derivative of mutual information with respect to this budget equals the minimum expected relative-entropy loss for estimating the input's output distribution. The posterior average of the channel's output distributions achieves this minimum and is the Bayesian predictor for the next observation. When the input distribution, the channel, or both are misspecified, the relative entropy between the true and assumed observation laws is the integral of the excess prediction loss over the budget. We also extend these relations to finite-state processes taking values in the input alphabet and observed through the channel at independent Poisson sampling times. Both in the matched setting and under model mismatch, the expected prediction loss accumulated over time using past observations equals the corresponding loss using the full observation record (noncausal estimation), averaged over lower sampling rates. We also characterize how redundancy between two outputs selects among input distributions that achieve one-use capacity. For the coupon-collector channel, where each observation is sampled uniformly from an unknown subset, we derive the exact capacity loss of any input distribution. In particular, balanced subset families can substantially reduce the input alphabet while preserving the leading capacity terms.
ER -