Timezone: »

 
Poster
Decentralized SGD and Average-direction SAM are Asymptotically Equivalent
Tongtian Zhu · Fengxiang He · Kaixuan Chen · Mingli Song · Dacheng Tao

Wed Jul 26 02:00 PM -- 03:30 PM (PDT) @ Exhibit Hall 1 #233
in Poster Session 3 »
Event URL: https://github.com/Raiden-Zhu/ICML-2023-DSGD-and-SAM »
Decentralized stochastic gradient descent (D-SGD) allows collaborative learning on massive devices simultaneously without the control of a central server. However, existing theories claim that decentralization invariably undermines generalization. In this paper, we challenge the conventional belief and present a completely new perspective for understanding decentralized learning. We prove that D-SGD implicitly minimizes the loss function of an average-direction Sharpness-aware minimization (SAM) algorithm under general non-convex non-$\beta$-smooth settings. This surprising asymptotic equivalence reveals an intrinsic regularization-optimization trade-off and three advantages of decentralization: (1) there exists a free uncertainty evaluation mechanism in D-SGD to improve posterior estimation; (2) D-SGD exhibits a gradient smoothing effect; and (3) the sharpness regularization effect of D-SGD does not decrease as total batch size increases, which justifies the potential generalization benefit of D-SGD over centralized SGD (C-SGD) in large-batch scenarios.
[ Slides]  [ PDF]  [ Poster] 

Author Information

Tongtian Zhu (Zhejiang University)
Fengxiang He (University of Edinburgh)

Fengxiang He is a Lecturer at Artificial Intelligence and its Applications Institute, School of Informatics, University of Edinburgh. He received his BSc in statistics from University of Science and Technology of China, MPhil and PhD in computer science from University of Sydney. He was an Algorithm Scientist at JD Explore Academy, JD.com, Inc., leading its trustworthy AI team. His research interest is in the theory and practice of trustworthy AI, including deep learning theory, privacy-preserving machine learning, algorithmic game theory, etc., as well as applications in finance and economics. He is an Area Chair of UAI, AISTATS, and ACML.

Kaixuan Chen (Zhejiang University)
Mingli Song (Zhejiang University)
Dacheng Tao

More from the Same Authors