Timezone: »
Oral
Global Convergence of Block Coordinate Descent in Deep Learning
Jinshan ZENG · Tim Tsz-Kit Lau · Shaobo Lin · Yuan Yao
Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature of DNN training. In this paper, we aim at providing a general methodology for provable convergence guarantees for this type of methods. In particular, for most of the commonly used DNN training models involving both two- and three-splitting schemes, we establish the global convergence to a critical point at a ${\cal O}(1/k)$ rate, where $k$ is the number of iterations. The results extend to general loss functions which have Lipschitz continuous gradients and deep residual networks (ResNets). Our key development adds several new elements to the Kurdyka-{\L}ojasiewicz inequality framework that enables us to carry out the global convergence analysis of BCD in the general scenario of deep learning.
Author Information
Jinshan ZENG (Hongkong University of Science and Technology)
Tim Tsz-Kit Lau (Northwestern University)
Shaobo Lin (Wenzhou University)
Yuan Yao (HongKong University of Science and Technology)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Global Convergence of Block Coordinate Descent in Deep Learning »
Fri Jun 14th 01:30 -- 04:00 AM Room Pacific Ballroom
More from the Same Authors
-
2022 Poster: Private Streaming SCO in $\ell_p$ geometry with Applications in High Dimensional Online Decision Making »
Yuxuan Han · Zhicong Liang · Zhipeng Liang · Yang Wang · Yuan Yao · Jiheng Zhang -
2022 Spotlight: Private Streaming SCO in $\ell_p$ geometry with Applications in High Dimensional Online Decision Making »
Yuxuan Han · Zhicong Liang · Zhipeng Liang · Yang Wang · Yuan Yao · Jiheng Zhang -
2020 Poster: DessiLBI: Exploring Structural Sparsity of Deep Networks via Differential Inclusion Paths »
Yanwei Fu · Chen Liu · Donghao Li · Xinwei Sun · Jinshan ZENG · Yuan Yao