Patnaik-Pearson intrinsic dimension for internal representations of neural networks
Abstract
We define a new measure of intrinsic dimension of a data manifold, which we call the Patnaik-Pearson dimension, and apply this to internal representations of neural networks, in particular transformers. The inspiration for this comes from the HTSR and SETOL work of Martin, Mahoney and Hinrichs, combined with the TwoNN intrinsic dimension estimator of Facco et al. We prove various properties of this intrinsic dimension estimator. Treating weight matrices of neural networks as data manifolds, for weight matrices whose empirical spectral density follows a Pareto (power law) distribution, we relate the Patnaik-Pearson dimension to the HTSR and SETOL analysis, and show that critical values of the tail exponent coincide for the two approaches. Using a combination of theoretical and numerical techniques, including Voiculescu's free probability, we study the behaviour of the Patnaik-Pearson dimension of a data manifold under the transformations typical to neural networks. We apply this machinery to the BERT-base and DeepSeek-R1-Distill-Qwen-1 models, to investigate first the Patnaik-Pearson dimension of the initial data manifold of token embeddings, and second the evolution of the Patnaik-Pearson dimension as token embeddings pass through the layers of the model. Code and notebooks used for the numerical results presented here is available at https://github.com/tdhadfield/PatnaikPearson
Related
- Formalizing the Intrinsic Dimension of Objective Landscapes by a Generalized Notion of Co-dimension
- Intrinsic-dimension empirical Bernstein inequalities for bounded self-adjoint operators
- Minimax Lower Bound for Estimating Diffusion-based Local Intrinsic Dimension
- Pairwise Distortion Distribution with Graph-Based Importance Sampling for Quantization Error and Intrinsic Dimension Estimation
- Ramified tori and Gelfand--Kirillov dimension of mod p representations
BibTeX
@misc{hadfield2026patnaikpearson,
title = {{Patnaik-Pearson intrinsic dimension for internal representations of neural networks}},
author = {Tom Hadfield},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.19268},
eprint = {2606.19268},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.19268},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Patnaik-Pearson intrinsic dimension for internal representations of neural networks
AU - Tom Hadfield
PY - 2026
DA - 2026/09/20
PB - arXiv
DO - 10.48550/arxiv.2606.19268
UR - https://arxiv.org/abs/2606.19268
AB - We define a new measure of intrinsic dimension of a data manifold, which we call the Patnaik-Pearson dimension, and apply this to internal representations of neural networks, in particular transformers. The inspiration for this comes from the HTSR and SETOL work of Martin, Mahoney and Hinrichs, combined with the TwoNN intrinsic dimension estimator of Facco et al. We prove various properties of this intrinsic dimension estimator. Treating weight matrices of neural networks as data manifolds, for weight matrices whose empirical spectral density follows a Pareto (power law) distribution, we relate the Patnaik-Pearson dimension to the HTSR and SETOL analysis, and show that critical values of the tail exponent coincide for the two approaches. Using a combination of theoretical and numerical techniques, including Voiculescu's free probability, we study the behaviour of the Patnaik-Pearson dimension of a data manifold under the transformations typical to neural networks. We apply this machinery to the BERT-base and DeepSeek-R1-Distill-Qwen-1 models, to investigate first the Patnaik-Pearson dimension of the initial data manifold of token embeddings, and second the evolution of the Patnaik-Pearson dimension as token embeddings pass through the layers of the model. Code and notebooks used for the numerical results presented here is available at https://github.com/tdhadfield/PatnaikPearson
ER -