Invariance before structure: two modes of emergent compression in representation learning
Abstract
Learned representations show striking invariance across biological and artificial neural networks, with different substrates and variations driven by drift, heterogeneity and initialisation. To understand this invariance, we propose Emergent Adaptive Representation Learning (EARL), a framework that distinguishes two modes of emergence in learned representations: state-space compression, where a learned representation gains predictive power over its own future beyond what the network's units can predict individually, and parameter-space compression, where networks with different micro-level parameters converge to the same representation. We formalise both as Fisher sensitivities and introduce ensemble-level information-theoretic measures for each. In a recurrent reservoir with a trained bottleneck, parameter-space compression rises within the first few epochs of training and partly independently of the task, whereas state-space compression rises about 30 epochs later and only when the network begins to learn the temporal structure of the task. These results suggest that invariance to the substrate and task-specific emergent structure are acquired in distinct phases of learning.
Related
- Emergent Generalization by Representation Learning in Artificial Neural Networks
- CUBE: Multimodal Representation Learning Reveals Biological Structure Across Histomorphology, Spatial Protein Phenotypes, and Transcriptome-Associated Signals
- Particle-based representation of phytoplankton modulates emergent coexistence and biogeography in a model ocean
- Emergent topological structure in spontaneous brain-organoid activity
- Abstract representation in the hippocampus predicts spontaneously adopted structure-based behavioral strategy
BibTeX
@misc{rajpal2026invariance,
title = {{Invariance before structure: two modes of emergent compression in representation learning}},
author = {Hardik Rajpal and Dan Goodman},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.07952},
eprint = {2611.07952},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.07952},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Invariance before structure: two modes of emergent compression in representation learning
AU - Hardik Rajpal
AU - Dan Goodman
PY - 2026
DA - 2026/11/05
PB - arXiv
DO - 10.48550/arxiv.2611.07952
UR - https://arxiv.org/abs/2611.07952
AB - Learned representations show striking invariance across biological and artificial neural networks, with different substrates and variations driven by drift, heterogeneity and initialisation. To understand this invariance, we propose Emergent Adaptive Representation Learning (EARL), a framework that distinguishes two modes of emergence in learned representations: state-space compression, where a learned representation gains predictive power over its own future beyond what the network's units can predict individually, and parameter-space compression, where networks with different micro-level parameters converge to the same representation. We formalise both as Fisher sensitivities and introduce ensemble-level information-theoretic measures for each. In a recurrent reservoir with a trained bottleneck, parameter-space compression rises within the first few epochs of training and partly independently of the task, whereas state-space compression rises about 30 epochs later and only when the network begins to learn the temporal structure of the task. These results suggest that invariance to the substrate and task-specific emergent structure are acquired in distinct phases of learning.
ER -