Entropy Estimation: From Statistical Physics to Classification
- 1Dipartimento di Scienze Matematiche, Informatiche e Fisiche (DMIF), University of Udine, Via delle Scienze 206, 33100 Udine, Italy
Abstract
Driven by the common entropy formalism in statistical physics and information theory, here we show that methods developed for the analysis of molecular configurational ensembles may be successfully used for classification. Entropy is estimated using the k-th nearest neighbor method combined with the Maximum Information Spanning Tree method to account for the mutual information between variables. This formalism is used to classify new samples using their estimated cross-entropy with the training set. The method is tested using the hand-written digits MNIST and the functional genomics CuMiDa datasets. The results obtained on these diverse datasets are comparable to those obtained by machine learning methods, or better when the number of samples is small. The methods described here have the advantage, compared to other classification methods, of directly linking variables and pairs of variables to the assigned class, a valuable feature when aiming at identifying causal relationships between features and classes (e.g., the relationship between genes and diseases).
Related
- Statistical Physics of Fish Collective Motion
- Statistical physics of language change inferred from time evolving maps
- Reference-guided defocus estimation enables physics-guided image restoration in deep ultraviolet wafer microscopy
- Optimal von Neumann Entropy Estimation
- Revisiting entropy and entropy-related magnitudes of a symbolic sequence and their estimation
BibTeX
@article{vello2026entropy,
title = {{Entropy Estimation: From Statistical Physics to Classification}},
author = {Federico Vello and Sara Fortuna and Federico Fogolari},
journal = {International Journal of Molecular Sciences},
year = {2026},
month = oct,
doi = {10.3390/ijms27209361},
url = {https://www.mdpi.com/1422-0067/27/20/9361},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Entropy Estimation: From Statistical Physics to Classification
AU - Federico Vello
AU - Sara Fortuna
AU - Federico Fogolari
PY - 2026
DA - 2026/10/21
JO - International Journal of Molecular Sciences
DO - 10.3390/ijms27209361
UR - https://www.mdpi.com/1422-0067/27/20/9361
AB - Driven by the common entropy formalism in statistical physics and information theory, here we show that methods developed for the analysis of molecular configurational ensembles may be successfully used for classification. Entropy is estimated using the k-th nearest neighbor method combined with the Maximum Information Spanning Tree method to account for the mutual information between variables. This formalism is used to classify new samples using their estimated cross-entropy with the training set. The method is tested using the hand-written digits MNIST and the functional genomics CuMiDa datasets. The results obtained on these diverse datasets are comparable to those obtained by machine learning methods, or better when the number of samples is small. The methods described here have the advantage, compared to other classification methods, of directly linking variables and pairs of variables to the assigned class, a valuable feature when aiming at identifying causal relationships between features and classes (e.g., the relationship between genes and diseases).
SN - 1422-0067
ER -