Optimization, Generalization and Differential Privacy Bounds for Gradient Descent on Kolmogorov–Arnold Networks

Kolmogorov--Arnold Networks (KANs) have recently emerged as a structured alternative to standard MLPs, yet a principled theory for their training dynamics, generalization, and privacy properties remains limited. In this paper, we analyze gradient descent (GD) for training two-layer KANs and derive general bounds that characterize their training dynamics, generalization, and utility under differential privacy (DP). As a concrete instantiation, we specialize our analysis to logistic loss under an NTK-separable assumption, where we show that polylogarithmic network width suffices for GD to achieve an optimization rate of order $1/T$ and a generalization rate of order $1/n$, with $T$ denoting the number of GD iterations and $n$ the sample size. In the private setting, we characterize the noise required for $(\epsilon,\delta)$-DP and obtain a utility bound of order $\sqrt{d}/(n\epsilon)$ (with $d$ the input dimension), matching the classical lower bound for general convex Lipschitz problems. Our results imply that polylogarithmic width is not only sufficient but also necessary under differential privacy, revealing a qualitative gap between non-private (sufficiency only) and private (necessity also emerges) training regimes. Experiments further illustrate how these theoretical insights can guide practical choices, including network width selection and early stopping.

Kolmogorov–Arnold Networks (KANs) are a recent neural network architecture that replaces fixed activation functions with learnable one-dimensional functions. They have shown promising performance in scientific and biomedical applications, but their training behavior is still not well understood. This paper studies when gradient descent can reliably train two-layer KANs, how well the trained models generalize to unseen data, and what happens when training must also protect data privacy. We prove that, under a standard separability condition, KANs can be trained successfully with only moderately large width, and that the resulting models achieve strong generalization guarantees. We also analyze a differentially private version of gradient descent, where noise is added during training to limit the influence of each individual data point. Our results quantify the trade-off between privacy and accuracy, and show how theory can guide practical choices such as network width and early stopping.