Formalizing the Intrinsic Dimension of Objective Landscapes by a Generalized Notion of Co-dimension
Abstract
In this work, we formalize the concept of intrinsic dimension of objective landscapes introduced in Li et al. (2018), which measures the structural complexity of learning problems by training along randomly generated subspaces. We define the intrinsic dimension of a set Θ⊂ℝp as the minimal dimension dint(Θ) such that a random plane of dimension dint(Θ) in ℝp intersects Θ with probability one. We show that it is a generalized notion of co-dimension that depends on the global geometry of Θ. We study mathematical properties of the intrinsic dimension and develop techniques to estimate the intrinsic dimension of sets Θ that arise in learning problems. More specifically, we derive upper and/or lower bounds where Θ is the set of optimal parameters in student-teacher settings with shallow and deep ReLU networks, including finite and infinite sample cases, and with and without regularization. We discuss how our theory leads to novel types of generalization bounds, thereby showing that the intrinsic dimension is a measure of learning complexity. Our results connect techniques from seemingly different areas of machine learning research such as classification of symmetries of ReLU networks and stochastic geometry.
Related
- Intrinsic-dimension empirical Bernstein inequalities for bounded self-adjoint operators
- Minimax Lower Bound for Estimating Diffusion-based Local Intrinsic Dimension
- Patnaik-Pearson intrinsic dimension for internal representations of neural networks
- Pairwise Distortion Distribution with Graph-Based Importance Sampling for Quantization Error and Intrinsic Dimension Estimation
- Manifold Hypothesis: Conditional Certificates and Consistent Intrinsic Dimension Estimation
BibTeX
@misc{araya2026formalizing,
title = {{Formalizing the Intrinsic Dimension of Objective Landscapes by a Generalized Notion of Co-dimension}},
author = {Ernesto Araya and Gitta Kutyniok and Weilin Li and Johannes Maly and Manjot Singh},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.39674},
eprint = {2610.39674},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.39674},
}
RIS
TY - UNPB
TI - Formalizing the Intrinsic Dimension of Objective Landscapes by a Generalized Notion of Co-dimension
AU - Ernesto Araya
AU - Gitta Kutyniok
AU - Weilin Li
AU - Johannes Maly
AU - Manjot Singh
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.39674
UR - https://arxiv.org/abs/2610.39674
AB - In this work, we formalize the concept of intrinsic dimension of objective landscapes introduced in Li et al. (2018), which measures the structural complexity of learning problems by training along randomly generated subspaces. We define the intrinsic dimension of a set Θ⊂ℝ^p as the minimal dimension d_int(Θ) such that a random plane of dimension d_int(Θ) in ℝ^p intersects Θ with probability one. We show that it is a generalized notion of co-dimension that depends on the global geometry of Θ. We study mathematical properties of the intrinsic dimension and develop techniques to estimate the intrinsic dimension of sets Θ that arise in learning problems. More specifically, we derive upper and/or lower bounds where Θ is the set of optimal parameters in student-teacher settings with shallow and deep ReLU networks, including finite and infinite sample cases, and with and without regularization. We discuss how our theory leads to novel types of generalization bounds, thereby showing that the intrinsic dimension is a measure of learning complexity. Our results connect techniques from seemingly different areas of machine learning research such as classification of symmetries of ReLU networks and stochastic geometry.
ER -