Timezone: »
Federated Learning (FL) is an emerging learning scheme that allows different distributed clients to train deep neural networks together without data sharing. Neural networks have become popular due to their unprecedented success. To the best of our knowledge, the theoretical guarantees of FL concerning neural networks with explicit forms and multi-step updates are unexplored. Nevertheless, training analysis of neural networks in FL is non-trivial for two reasons: first, the objective loss function we are optimizing is non-smooth and non-convex, and second, we are even not updating in the gradient direction. Existing convergence results for gradient descent-based methods heavily rely on the fact that the gradient direction is used for updating. The current paper presents a new class of convergence analysis for FL, Federated Neural Tangent Kernel (FL-NTK), which corresponds to overparamterized ReLU neural networks trained by gradient descent in FL and is inspired by the analysis in Neural Tangent Kernel (NTK). Theoretically, FL-NTK converges to a global-optimal solution at a linear rate with properly tuned learning parameters. Furthermore, with proper distributional assumptions, FL-NTK can also achieve good generalization. The proposed theoretical analysis scheme can be generalized to more complex neural networks.
Author Information
Baihe Huang (Peking University)
Xiaoxiao Li (The University of British Columbia)
Zhao Song (UT-Austin & University of Washington)
Xin Yang (University of Washington)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Spotlight: FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Analysis »
Thu. Jul 22nd 12:30 -- 12:35 AM Room
More from the Same Authors
-
2021 : BrainNNExplainer: An Interpretable Graph Neural Network Framework for Brain Network based Disease Analysis »
Hejie Cui · Wei Dai · Yanqiao Zhu · Xiaoxiao Li · Lifang He · Carl Yang -
2021 : One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images »
Weina Jin · Xiaoxiao Li · Ghassan Hamarneh -
2021 : One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images »
Weina Jin · Xiaoxiao Li · Ghassan Hamarneh -
2023 : H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models »
Zhenyu Zhang · Ying Sheng · Tianyi Zhou · Tianlong Chen · Lianmin Zheng · Ruisi Cai · Zhao Song · Yuandong Tian · Christopher Re · Clark Barrett · Zhangyang “Atlas” Wang · Beidi Chen -
2023 Oral: Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time »
Zichang Liu · Jue Wang · Tri Dao · Tianyi Zhou · Binhang Yuan · Zhao Song · Anshumali Shrivastava · Ce Zhang · Yuandong Tian · Christopher Re · Beidi Chen -
2023 Poster: Federated Adversarial Learning: A Framework with Convergence Analysis »
Xiaoxiao Li · Zhao Song · Jiaming Yang -
2023 Poster: Sketching for First Order Method: Efficient Algorithm for Low-Bandwidth Channel and Vulnerability »
Zhao Song · Yitan Wang · Zheng Yu · Lichen Zhang -
2023 Poster: A Nearly-Optimal Bound for Fast Regression with $\ell_\infty$ Guarantee »
Zhao Song · Mingquan Ye · Junze Yin · Lichen Zhang -
2023 Poster: Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time »
Zichang Liu · Jue Wang · Tri Dao · Tianyi Zhou · Binhang Yuan · Zhao Song · Anshumali Shrivastava · Ce Zhang · Yuandong Tian · Christopher Re · Beidi Chen -
2023 Poster: Sketching Meets Differential Privacy: Fast Algorithm for Dynamic Kronecker Projection Maintenance »
Zhao Song · Xin Yang · Yuanyuan Yang · Lichen Zhang -
2021 : Closing remarks »
Xiaoxiao Li -
2021 Workshop: Interpretable Machine Learning in Healthcare »
Yuyin Zhou · Xiaoxiao Li · Vicky Yao · Pengtao Xie · DOU QI · Nicha Dvornek · Julia Schnabel · Judy Wawira · Yifan Peng · Ronald Summers · Alan Karthikesalingam · Lei Xing · Eric Xing -
2021 Poster: Fast Sketching of Polynomial Kernels of Polynomial Degree »
Zhao Song · David Woodruff · Zheng Yu · Lichen Zhang -
2021 Spotlight: Fast Sketching of Polynomial Kernels of Polynomial Degree »
Zhao Song · David Woodruff · Zheng Yu · Lichen Zhang -
2021 Poster: Oblivious Sketching-based Central Path Method for Linear Programming »
Zhao Song · Zheng Yu -
2021 Spotlight: Oblivious Sketching-based Central Path Method for Linear Programming »
Zhao Song · Zheng Yu -
2020 Poster: Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE »
Juntang Zhuang · Nicha Dvornek · Xiaoxiao Li · Sekhar Tatikonda · Xenophon Papademetris · James Duncan -
2020 Poster: Meta-learning for Mixed Linear Regression »
Weihao Kong · Raghav Somani · Zhao Song · Sham Kakade · Sewoong Oh