Principal Component Regression Dominates all Monotone Spectral Filters for Linear Regression

Abstract

We compare the instance-wise, finite-sample risks of monotone spectral filters for linear regression, a broad class of estimators including principal component regression (PCR), gradient descent (GD), and ridge regression. We show that PCR dominates all monotone spectral filters: compared to any such filter, the risk of optimally tuned PCR is no bigger by a constant factor for all problems. Furthermore, the dominance is strong if the filter is separated from step functions (e.g., GD and ridge): there exist problem instances for which the risk of PCR is smaller by a polynomial factor in sample size dependence. Our comparison results show that PCR is optimal and thus admissible among monotone filters, significantly extending Wu et al. (2026)'s result that GD strongly dominates ridge. From a technical perspective, we establish new upper and lower bounds for general spectral filters, which are instance-wise sharp when specialized to ridge or GD, recovering or improving the best-known bounds.

Variable-Projection Sparse Functional Principal Component Analysis: Interpretable Functional Dimensionality Reduction with Applications to Raman Spectral Data
Han Ying Lim, Dharini Pathmanathan, Philipp Otto +1
arXiv16 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv
Principal component analysis in Φ _α-linear spaces
Cassandra Mussard, Walter Briec, Charles Condevaux +1
Journal of Big Data24 May 2026Peer-reviewedCC BY-NC-NDStatisticsHAL
Robust Multi-Task Learning for Principal Component Analysis
Dali Liu, Haolei Weng
arXiv17 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv
Functional Principal Component Analysis for Manifold-Indexed Data
Chang Jun Im, Jeong Min Jeon
arXiv13 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Robust Principal Component Analysis via ℓ₀ Alternating Minimization
Ze-Kun Lyu, Xin Liu, Qian-Yu Shu +2
arXiv30 Oct 2026PreprintCC BY 4.0Full textOptimisation & controlarXiv

Cite

BibTeX RIS

BibTeX

@misc{kim2026principal,
  title = {{Principal Component Regression Dominates all Monotone Spectral Filters for Linear Regression}},
  author = {Juno Kim and Hengyu Fu and Peter Bartlett and Jason D. Lee and Jingfeng Wu},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.39440},
  eprint = {2609.39440},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.39440},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Principal Component Regression Dominates all Monotone Spectral Filters for Linear Regression
AU  - Juno Kim
AU  - Hengyu Fu
AU  - Peter Bartlett
AU  - Jason D. Lee
AU  - Jingfeng Wu
PY  - 2026
DA  - 2026/09/30
PB  - arXiv
DO  - 10.48550/arxiv.2609.39440
UR  - https://arxiv.org/abs/2609.39440
AB  - We compare the instance-wise, finite-sample risks of monotone spectral filters for linear regression, a broad class of estimators including principal component regression (PCR), gradient descent (GD), and ridge regression. We show that PCR dominates all monotone spectral filters: compared to any such filter, the risk of optimally tuned PCR is no bigger by a constant factor for all problems. Furthermore, the dominance is strong if the filter is separated from step functions (e.g., GD and ridge): there exist problem instances for which the risk of PCR is smaller by a polynomial factor in sample size dependence. Our comparison results show that PCR is optimal and thus admissible among monotone filters, significantly extending Wu et al. (2026)'s result that GD strongly dominates ridge. From a technical perspective, we establish new upper and lower bounds for general spectral filters, which are instance-wise sharp when specialized to ridge or GD, recovering or improving the best-known bounds.
ER  -

Abstract

Related

Cite