Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy
Abstract
Overparameterized neural networks carry far more hidden units than a task nominally requires, raising the question of which neurons are essential and whether that distinction is legible in the representation itself, without labels or gradients. We cast neuron selection as the problem of coarse-graining the hidden layer by retaining a subset of its neurons, and score each putative selection by the mapping entropy (ME). This quantity measures the loss of discriminatory power inherent in discarding part of the network neurons, and the selection that minimises the ME is taken as particularly informative. This criterion is fully unsupervised, in that it depends only on hidden-activation statistics. In teacher-student networks, ME optimisation recovers the minimal teacher-consistent representation and retains extra units in proportion to the hidden layer's residual variability; in a non-linear Gaussian process task, it selects coherent functional-class mappings whose preferred class shifts across training. On this task and on translation-augmented MNIST, ME-selected subnetworks outperform random subsets of equal size, most clearly under strong compression - linking configurational distinguishability to predictive performance.
Related
- Bliss of dimensionality: How an unsupervised criterion identifies optimal low-resolution representations of high-dimensional datasets
- Hierarchical Unsupervised Clustering of Nanobeam Electron Diffraction Data Using Sparse Bragg-Vector Representations
- Pareto-optimal quantum kernel selection for unsupervised anomaly detection on real malware beaconing data
- Relative entropy representations via tracial joint spectral measures
- Mapping Schwarzschild Spacetime: From Kruskal to Diamond and Golden Representations
BibTeX
@misc{mele2026coarsegraining,
title = {{Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy}},
author = {Margherita Mele and Andrea Castagna and Roberto Menichetti and Raffaello Potestio and Alessandro Ingrosso},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.05126},
eprint = {2609.05126},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.05126},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy
AU - Margherita Mele
AU - Andrea Castagna
AU - Roberto Menichetti
AU - Raffaello Potestio
AU - Alessandro Ingrosso
PY - 2026
DA - 2026/09/04
PB - arXiv
DO - 10.48550/arxiv.2609.05126
UR - https://arxiv.org/abs/2609.05126
AB - Overparameterized neural networks carry far more hidden units than a task nominally requires, raising the question of which neurons are essential and whether that distinction is legible in the representation itself, without labels or gradients. We cast neuron selection as the problem of coarse-graining the hidden layer by retaining a subset of its neurons, and score each putative selection by the mapping entropy (ME). This quantity measures the loss of discriminatory power inherent in discarding part of the network neurons, and the selection that minimises the ME is taken as particularly informative. This criterion is fully unsupervised, in that it depends only on hidden-activation statistics. In teacher-student networks, ME optimisation recovers the minimal teacher-consistent representation and retains extra units in proportion to the hidden layer's residual variability; in a non-linear Gaussian process task, it selects coherent functional-class mappings whose preferred class shifts across training. On this task and on translation-augmented MNIST, ME-selected subnetworks outperform random subsets of equal size, most clearly under strong compression - linking configurational distinguishability to predictive performance.
ER -