Optimization Risk Bounds for Kolmogorov-Arnold Networks Trained by DP-SGD with Correlated Noise
Abstract
The theoretical understanding of differentially private stochastic gradient descent (DP-SGD) with temporally correlated noise remains limited, particularly for non-convex neural network training. As a first step, we study two-layer Kolmogorov-Arnold Networks (KANs), a recently introduced architecture with learnable spline-based edge functions. We establish the first optimization risk bounds for clipped mini-batch DP-SGD with correlated noise in this setting, with explicit dependence on temporal correlation, clipping, mini-batch sampling, and network width. Existing arguments fail for three reasons: temporal dependence breaks the conditional-centering step; projection obstructs the cross-iteration cancellation of correlated perturbations; and active clipping breaks the empirical-gradient structure. We address these issues through shifted and auxiliary dynamics, a weighted empirical loss, and a high-probability localization argument. Our bound shows that temporal correlation reduces the leading noise terms, the clipping threshold enters essentially through an effective step size, and private training admits an explicit network width range. Experiments on synthetic data and MNIST support the predicted optimization effect of temporal correlation. As an application, we derive population risk guarantees via algorithmic stability. Our framework recovers non-private mini-batch SGD, independent-noise DP-SGD, and their full-batch counterparts as special cases.
Related
- Kolmogorov-Arnold Networks for Free-Boundary Partial Differential Equations
- AKAPINN: Adaptive Kolmogorov-Arnold Physics-Informed Neural Networks for approximating solutions to quasilinear partial differential equations
- Geometry-Constrained Kolmogorov-Arnold Networks: Learning Edge Geometry via Banach Duality
- Tractable Inference with Kolmogorov-Arnold Networks
- Mesh-Free Numerical Approximation of the Biharmonic Equation via Optimized Kolmogorov--Arnold Neural Networks
BibTeX
@misc{wang2026optimization,
title = {{Optimization Risk Bounds for Kolmogorov-Arnold Networks Trained by DP-SGD with Correlated Noise}},
author = {Puyu Wang and Jan Schuchardt and Nikita Kalinin and Marcio Monteiro and Junyu Zhou and Sophie Fellenz and Christoph Lampert and Marius Kloft},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.12648},
eprint = {2605.12648},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.12648},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Optimization Risk Bounds for Kolmogorov-Arnold Networks Trained by DP-SGD with Correlated Noise
AU - Puyu Wang
AU - Jan Schuchardt
AU - Nikita Kalinin
AU - Marcio Monteiro
AU - Junyu Zhou
AU - Sophie Fellenz
AU - Christoph Lampert
AU - Marius Kloft
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2605.12648
UR - https://arxiv.org/abs/2605.12648
AB - The theoretical understanding of differentially private stochastic gradient descent (DP-SGD) with temporally correlated noise remains limited, particularly for non-convex neural network training. As a first step, we study two-layer Kolmogorov-Arnold Networks (KANs), a recently introduced architecture with learnable spline-based edge functions. We establish the first optimization risk bounds for clipped mini-batch DP-SGD with correlated noise in this setting, with explicit dependence on temporal correlation, clipping, mini-batch sampling, and network width. Existing arguments fail for three reasons: temporal dependence breaks the conditional-centering step; projection obstructs the cross-iteration cancellation of correlated perturbations; and active clipping breaks the empirical-gradient structure. We address these issues through shifted and auxiliary dynamics, a weighted empirical loss, and a high-probability localization argument. Our bound shows that temporal correlation reduces the leading noise terms, the clipping threshold enters essentially through an effective step size, and private training admits an explicit network width range. Experiments on synthetic data and MNIST support the predicted optimization effect of temporal correlation. As an application, we derive population risk guarantees via algorithmic stability. Our framework recovers non-private mini-batch SGD, independent-noise DP-SGD, and their full-batch counterparts as special cases.
ER -