Learning reshapes power-law anisotropy in internal representations
Abstract
Power-law anisotropy in internal representations has been observed across a wide range of biological and artificial neural systems, from state-of-the-art language models to the mouse cerebral cortex. This anisotropy is a key geometric property of high-dimensional information processing and underlies a variety of theoretical analyses. However, the mechanism by which it emerges from input structure and task-driven learning has remained unclear. Here, we characterize this formation process by exactly solving the learning dynamics of a wide two-layer linear neural network in a teacher--student setting with power-law input and teacher structures. We show that, in the feature-learning regime, the local power-law exponent of the internal-representation spectrum evolves nonmonotonically over the course of training and exhibits up to four distinct asymptotic regimes across modes and training times. By contrast, in the lazy regime, the exponent remains essentially unchanged. We further demonstrate numerically that similar exponent dynamics arise in more realistic nonlinear networks. Together, these results suggest a general mechanism by which the dynamic interaction between input statistics and task structure gives rise to power-law internal representations.
Related
- Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy
- Patnaik-Pearson intrinsic dimension for internal representations of neural networks
- Tail representations for absolute, signed-power, inverse, and logarithmic moments
- Kleisli convolution representations of power monoids
- Comparison of neural and spline representations for physics-informed learning
BibTeX
@misc{nakamuta2026learning,
title = {{Learning reshapes power-law anisotropy in internal representations}},
author = {Asahi Nakamuta and Jun-nosuke Teramae},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.15239},
eprint = {2608.15239},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.15239},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Learning reshapes power-law anisotropy in internal representations
AU - Asahi Nakamuta
AU - Jun-nosuke Teramae
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2608.15239
UR - https://arxiv.org/abs/2608.15239
AB - Power-law anisotropy in internal representations has been observed across a wide range of biological and artificial neural systems, from state-of-the-art language models to the mouse cerebral cortex. This anisotropy is a key geometric property of high-dimensional information processing and underlies a variety of theoretical analyses. However, the mechanism by which it emerges from input structure and task-driven learning has remained unclear. Here, we characterize this formation process by exactly solving the learning dynamics of a wide two-layer linear neural network in a teacher--student setting with power-law input and teacher structures. We show that, in the feature-learning regime, the local power-law exponent of the internal-representation spectrum evolves nonmonotonically over the course of training and exhibits up to four distinct asymptotic regimes across modes and training times. By contrast, in the lazy regime, the exponent remains essentially unchanged. We further demonstrate numerically that similar exponent dynamics arise in more realistic nonlinear networks. Together, these results suggest a general mechanism by which the dynamic interaction between input statistics and task structure gives rise to power-law internal representations.
ER -