Selective Inference for Clustering Under Unknown Covariance
Abstract
Unsupervised clustering is widely used to discover underlying group structure, after which researchers often seek to validate these groups via post-hoc hypothesis testing on the population means. However, using the same data both to form clusters and to test for a difference in means inflates the Type I error rate. Recent work in selective inference has proposed solutions when the underlying data are drawn from a multivariate Gaussian distribution with a known covariance matrix or an unknown isotropic covariance matrix. We propose a new selective test for the difference in means between two clusters that provably controls the Type I error rate under an arbitrary unknown covariance matrix. Our test statistic does not require estimating the feature covariance matrix; instead, it leverages a Monte Carlo sampling procedure over the set of orthogonal matrices. We show that this new selective test can be computed efficiently, and we derive closed-form solutions when clusters are formed using k-means or hierarchical clustering. In simulations, we demonstrate that our proposed p-value controls the selective Type I error rate and achieves substantial power, and we illustrate its application on gene expression and drug review data sets.
Related
- Selective Inference for Deep Clustering in Latent Spaces
- Exact Selective Inference for Spectral Clustering via Piecewise-Constant Graph Perturbation
- Mixture-of-Finite-Mixtures Wishart Model for Clustering Covariance Matrices with an Application to Brain Functional Connectivity
- Federated Clustering with Unknown Local and Global Cluster Cardinalities
- Fast Functional Clustering via Closed-Form Covariance Updates
BibTeX
@misc{chen2026selective,
title = {{Selective Inference for Clustering Under Unknown Covariance}},
author = {Yiqun T. Chen and Lucy L. Gao},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.17458},
eprint = {2610.17458},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.17458},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Selective Inference for Clustering Under Unknown Covariance
AU - Yiqun T. Chen
AU - Lucy L. Gao
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2610.17458
UR - https://arxiv.org/abs/2610.17458
AB - Unsupervised clustering is widely used to discover underlying group structure, after which researchers often seek to validate these groups via post-hoc hypothesis testing on the population means. However, using the same data both to form clusters and to test for a difference in means inflates the Type I error rate. Recent work in selective inference has proposed solutions when the underlying data are drawn from a multivariate Gaussian distribution with a known covariance matrix or an unknown isotropic covariance matrix. We propose a new selective test for the difference in means between two clusters that provably controls the Type I error rate under an arbitrary unknown covariance matrix. Our test statistic does not require estimating the feature covariance matrix; instead, it leverages a Monte Carlo sampling procedure over the set of orthogonal matrices. We show that this new selective test can be computed efficiently, and we derive closed-form solutions when clusters are formed using k-means or hierarchical clustering. In simulations, we demonstrate that our proposed p-value controls the selective Type I error rate and achieves substantial power, and we illustrate its application on gene expression and drug review data sets.
ER -