SENECA: Small-Sample Discrete Entropy Estimation via Self-Consistent Missing Mass
Abstract
Discrete entropy estimation is a classic information theory problem, wherein the average information content of a discrete random variable is estimated from samples alone. Naive approaches, such as the plugin method, fail to account for the probability mass associated with members of the random variable's support that are unobserved in a given sample, known as the "missing mass." The resulting systemic underestimation is particularly problematic when data is time-consuming or costly to gather. We propose SENECA, an entropy estimation scheme based on a novel "self-consistent" missing mass calculation. Extensive numerical experiments indicate that our approach outperforms many state-of-the-art alternatives overall in the small-sample setting. We then apply SENECA to two practical use cases, namely biodiversity estimation and the detection of incorrect large language model responses, where our method is competitive with domain-specific approaches. Our work advances SENECA as an effective drop-in replacement for small-sample entropy estimation, with broad utility across several domains.
Related
- Fourth-Order Energy-Consistent Average Vector Field Discrete Gradient Methods for Dissipative Hamiltonian Systems
- Identification and Estimation of Causal Estimands with Missing Not at Random Data
- Model-based estimation and imputation with torus missing values
- Simulation-consistent Estimation of the Marginal Likelihood for Block Models
- Strongly Consistent Estimation of the Extended ℓ1-Sum of ϕ-Mixing Coefficients from a Single Trajectory
BibTeX
@misc{mccabe2026seneca,
title = {{SENECA: Small-Sample Discrete Entropy Estimation via Self-Consistent Missing Mass}},
author = {Lucas H. McCabe and H. Howie Huang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.00668},
eprint = {2605.00668},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.00668},
}
RIS
TY - UNPB
TI - SENECA: Small-Sample Discrete Entropy Estimation via Self-Consistent Missing Mass
AU - Lucas H. McCabe
AU - H. Howie Huang
PY - 2026
DA - 2026/10/25
PB - arXiv
DO - 10.48550/arxiv.2605.00668
UR - https://arxiv.org/abs/2605.00668
AB - Discrete entropy estimation is a classic information theory problem, wherein the average information content of a discrete random variable is estimated from samples alone. Naive approaches, such as the plugin method, fail to account for the probability mass associated with members of the random variable's support that are unobserved in a given sample, known as the "missing mass." The resulting systemic underestimation is particularly problematic when data is time-consuming or costly to gather. We propose SENECA, an entropy estimation scheme based on a novel "self-consistent" missing mass calculation. Extensive numerical experiments indicate that our approach outperforms many state-of-the-art alternatives overall in the small-sample setting. We then apply SENECA to two practical use cases, namely biodiversity estimation and the detection of incorrect large language model responses, where our method is competitive with domain-specific approaches. Our work advances SENECA as an effective drop-in replacement for small-sample entropy estimation, with broad utility across several domains.
ER -