Nonparametric Identification of Latent Dimension under Heavy-Tailed and Mixed-Signals
Abstract
Principal component analysis and factor analysis are foundational to the study of high-dimensional data, yet their efficacy depends entirely on correctly identifying the latent dimension r. While parallel analysis (PA) is widely regarded as the gold standard for this task, its performance suffers in high-dimensional regimes consisting of mixed signals and/or heavy-tailed error distributions. This arises because under the preceding conditions, column-wise permutation mechanism (or any modern variations) applied to the data matrix X fails to destroy the signal, leading to an inflated null spectrum. To address this, we propose a nonparametric solution. We show that the latent dimension is uniquely preserved in the degree of order within the rank-transformed data matrix, which restores spectral exchangeability under permutation. We prove the consistency of Rank PA under relaxed conditions, requiring neither the pervasiveness of factors nor restrictive fourth moment assumptions. We establish that Rank PA can consistently estimate the number of factors up to the maximum of O(n1/3 − τ) where τ>0. The robust performance of our remedy is demonstrated through numerical simulations and an analysis of the FRED-MD dataset.
Related
- Dimension-Corrected Hitting Times for Heavy-Tailed Spectral Emergence in Neural Optimizer Dynamics
- Identification of forward models: a nonparametric approach
- Dimension reduction of compositional data through mixtures of latent source compositions
- Understanding Latent-Dimension Scaling in Dynamical-System Learning through Spectral Reliability
- Common Signals, Spurious Cross-Excitation, and Identification in Multivariate Hawkes Processes
BibTeX
@misc{jha2026nonparametric,
title = {{Nonparametric Identification of Latent Dimension under Heavy-Tailed and Mixed-Signals}},
author = {Chetkar Jha},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.00588},
eprint = {2610.00588},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.00588},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Nonparametric Identification of Latent Dimension under Heavy-Tailed and Mixed-Signals
AU - Chetkar Jha
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.00588
UR - https://arxiv.org/abs/2610.00588
AB - Principal component analysis and factor analysis are foundational to the study of high-dimensional data, yet their efficacy depends entirely on correctly identifying the latent dimension r. While parallel analysis (PA) is widely regarded as the gold standard for this task, its performance suffers in high-dimensional regimes consisting of mixed signals and/or heavy-tailed error distributions. This arises because under the preceding conditions, column-wise permutation mechanism (or any modern variations) applied to the data matrix X fails to destroy the signal, leading to an inflated null spectrum. To address this, we propose a nonparametric solution. We show that the latent dimension is uniquely preserved in the degree of order within the rank-transformed data matrix, which restores spectral exchangeability under permutation. We prove the consistency of Rank PA under relaxed conditions, requiring neither the pervasiveness of factors nor restrictive fourth moment assumptions. We establish that Rank PA can consistently estimate the number of factors up to the maximum of O(n^1/3 − τ) where τ>0. The robust performance of our remedy is demonstrated through numerical simulations and an analysis of the FRED-MD dataset.
ER -