On Kernel Eigen-alignments of KRR: Reconstruction and Generalization
Abstract
This paper investigates the critical role of eigenalignments between the kernel matrix and learning targets in achieving robust generalization in learning problems. We establish a direct connection between generalization performance in kernel methods and the estimation of eigenvectors and eigenvalues of matrices, offering a more intuitive understanding compared to prior work with minimal assumptions. We also show that, since the prediction task in KRR is essentially the weighted sum of eigenvectors/singular vectors, by analyzing how much error can be caused by perturbations to the kernel matrix, we can then derive a bound on this generalization error using the estimation stability of matrix eigenvalues and eigenvectors. Compared with previous work, our analysis concentrates on finite-sample settings and on the generalization error arising from having a suboptimal finite training set. Our findings reveal that in kernel methods, as long as the kernel is of high rank, the near-zero reconstruction error can be trivially obtained, implying that the reconstruction error will have limited predictive power for generalization. Additionally, we establish a generalization bound from an eigenvalues/eigenvectors estimation perspective, showing that strong generalization requires increasing eigenvector alignment, eigenvalue magnitude, or gaps between consecutive eigenvalues. Finally, we also demonstrate empirically our simplified risk estimator is able to track testing errors in trailing eigendirections more closely than the existing literature.
Related
- Generalization Analysis of Distributed Kernel-based Robust Gradient Descent Algorithms
- Mask Reconstruction via Kernel-Based Surrogate Modeling for Automated Circulator Design
- Prescribed Beurling densities in spectral eigen-subspaces
- Essential-Surface Complexes of Knot Exteriors: Image, Kernel, and Reconstruction
- Families of relative periodic orbits in the planar three-body problem via consecutive alignments
BibTeX
@misc{liu2026kernel,
title = {{On Kernel Eigen-alignments of KRR: Reconstruction and Generalization}},
author = {Yang Liu and Ernest Fokoue and Richard Lange and Daniel Krutz},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.15240},
eprint = {2605.15240},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.15240},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - On Kernel Eigen-alignments of KRR: Reconstruction and Generalization
AU - Yang Liu
AU - Ernest Fokoue
AU - Richard Lange
AU - Daniel Krutz
PY - 2026
DA - 2026/09/19
PB - arXiv
DO - 10.48550/arxiv.2605.15240
UR - https://arxiv.org/abs/2605.15240
AB - This paper investigates the critical role of eigenalignments between the kernel matrix and learning targets in achieving robust generalization in learning problems. We establish a direct connection between generalization performance in kernel methods and the estimation of eigenvectors and eigenvalues of matrices, offering a more intuitive understanding compared to prior work with minimal assumptions. We also show that, since the prediction task in KRR is essentially the weighted sum of eigenvectors/singular vectors, by analyzing how much error can be caused by perturbations to the kernel matrix, we can then derive a bound on this generalization error using the estimation stability of matrix eigenvalues and eigenvectors. Compared with previous work, our analysis concentrates on finite-sample settings and on the generalization error arising from having a suboptimal finite training set. Our findings reveal that in kernel methods, as long as the kernel is of high rank, the near-zero reconstruction error can be trivially obtained, implying that the reconstruction error will have limited predictive power for generalization. Additionally, we establish a generalization bound from an eigenvalues/eigenvectors estimation perspective, showing that strong generalization requires increasing eigenvector alignment, eigenvalue magnitude, or gaps between consecutive eigenvalues. Finally, we also demonstrate empirically our simplified risk estimator is able to track testing errors in trailing eigendirections more closely than the existing literature.
ER -