A Framework for Supervised and Unsupervised Learning via Reproducing Kernel Hilbert Spaces

Abstract

We introduce a unified framework for supervised and unsupervised learning through density estimation using reproducing kernel Hilbert spaces (RKHSs). To this end, we introduce the notion of a nested RKHS, that is, an increasing sequence of RKHSs whose union is dense in an ambient L²-space. This construction allows us to exploit the computational structure of RKHS methods at each finite approximation level while preserving the approximation power of the entire L²-space. We first study statistical learning in a fixed RKHS. In both unsupervised and supervised settings, we establish strong convergence results for empirical averages of random elements taking values in an RKHS continuously embedded in L². We then use the nested RKHS structure to combine these statistical convergence results with the density of the union of the RKHSs in the ambient L²-space. We apply this construction to probability density estimation and binary Bayesian classification. For density estimation, we construct explicit empirical estimators of the orthogonal projections of the target density onto the RKHSs and establish their almost sure convergence. A diagonal selection procedure then yields a sequence of estimators converging almost surely to the target density in L². For binary Bayesian classification, we formulate the approximation of the Bayes target as the minimization of suitable population and empirical risk functionals. We prove consistency of the empirical minimizers and use the nested structure to obtain convergence towards the Bayes target in L². We further provide a geometric interpretation in terms of hyperplanes and weighted distances in the ambient L²-space. Finally, we show that the population risk admits a direct probabilistic interpretation in terms of the probability of misclassification.

Norm-optimal model order reduction in reproducing kernel Hilbert spaces
Tamás Dózsa, Andrea Angino, Rolf Krause +1
arXiv20 Oct 2026PreprintCC BY 4.0Full textNumerical analysis & scientific computingarXiv
Multiplicative linear functionals on reproducing kernel Hilbert spaces
Tirthankar Bhattacharyya, Jaikishan, Poornendu Kumar +1
arXiv2 Sep 2026PreprintarXiv licenceFull textAnalysis & PDEsarXiv
DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing
Sagnik Nandy, Samriddha Lahiry, Pragya Sur +1
arXiv25 Sep 2026PreprintCC BY 4.0Full textStatisticsarXiv
Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning
Joseph Egan, Ciarán M. Gilligan-Lee, Michael O'Riordan +1
arXiv9 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Joint Contrastive-Adversarial Representation Learning for Unsupervised Domain Adaptation
Yuling Jiao, Lican Kang, Jerry Zhijian Yang +2
arXiv11 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv

Cite

BibTeX RIS

BibTeX

@misc{protin2026framework,
  title = {{A Framework for Supervised and Unsupervised Learning via Reproducing Kernel Hilbert Spaces}},
  author = {Frédéric Protin},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2609.20792},
  eprint = {2609.20792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.20792},
}

RIS

TY  - UNPB
TI  - A Framework for Supervised and Unsupervised Learning via Reproducing Kernel Hilbert Spaces
AU  - Frédéric Protin
PY  - 2026
DA  - 2026/10/27
PB  - arXiv
DO  - 10.48550/arxiv.2609.20792
UR  - https://arxiv.org/abs/2609.20792
AB  - We introduce a unified framework for supervised and unsupervised learning through density estimation using reproducing kernel Hilbert spaces (RKHSs). To this end, we introduce the notion of a nested RKHS, that is, an increasing sequence of RKHSs whose union is dense in an ambient L^2-space. This construction allows us to exploit the computational structure of RKHS methods at each finite approximation level while preserving the approximation power of the entire L^2-space. We first study statistical learning in a fixed RKHS. In both unsupervised and supervised settings, we establish strong convergence results for empirical averages of random elements taking values in an RKHS continuously embedded in L^2. We then use the nested RKHS structure to combine these statistical convergence results with the density of the union of the RKHSs in the ambient L^2-space. We apply this construction to probability density estimation and binary Bayesian classification. For density estimation, we construct explicit empirical estimators of the orthogonal projections of the target density onto the RKHSs and establish their almost sure convergence. A diagonal selection procedure then yields a sequence of estimators converging almost surely to the target density in L^2. For binary Bayesian classification, we formulate the approximation of the Bayes target as the minimization of suitable population and empirical risk functionals. We prove consistency of the empirical minimizers and use the nested structure to obtain convergence towards the Bayes target in L^2. We further provide a geometric interpretation in terms of hyperplanes and weighted distances in the ambient L^2-space. Finally, we show that the population risk admits a direct probabilistic interpretation in terms of the probability of misclassification.
ER  -

Abstract

Related

Cite