Timezone: »
We propose Zeno++, a new robust asynchronous Stochastic Gradient Descent(SGD) procedure, intended to tolerate Byzantine failures of workers. In contrast to previous work, Zeno++ removes several unrealistic restrictions on worker-server communication, now allowing for fully asynchronous updates from anonymous workers, for arbitrarily stale worker updates, and for the possibility of an unbounded number of Byzantine workers. The key idea is to estimate the descent of the loss value after the candidate gradient is applied, where large descent values indicate that the update results in optimization progress. We prove the convergence of Zeno++ for non-convex problems under Byzantine failures. Experimental results show that Zeno++ outperforms existing Byzantine-tolerant asynchronous SGD algorithms.
Author Information
Cong Xie (UIUC)
Oluwasanmi Koyejo (Illinois / Google)

Sanmi (Oluwasanmi) Koyejo is an Assistant Professor in the Department of Computer Science at Stanford University. Koyejo was previously an Associate Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign. Koyejo's research interests are in developing the principles and practice of trustworthy machine learning, focusing on applications to neuroscience and healthcare. Koyejo completed a Ph.D. in Electrical Engineering at the University of Texas at Austin, advised by Joydeep Ghosh, and postdoctoral research at Stanford University with Russell A. Poldrack and Pradeep Ravikumar. Koyejo has been the recipient of several awards, including a best paper award from the conference on uncertainty in artificial intelligence, a Skip Ellis Early Career Award, a Sloan Fellowship, a Terman faculty fellowship, an NSF CAREER award, a Kavli Fellowship, an IJCAI early career spotlight, and a trainee award from the Organization for Human Brain Mapping. Koyejo spends time at Google as a part of the Brain team, serves on the Neural Information Processing Systems Foundation Board, the Association for Health Learning and Inference Board, and as president of the Black in AI organization.
Indranil Gupta (UIUC)
More from the Same Authors
-
2022 : Adapting to Shifts in Latent Confounders via Observed Concepts and Proxies »
Matt Kusner · Ibrahim Alabdulmohsin · Stephen Pfohl · Olawale Salaudeen · Arthur Gretton · Sanmi Koyejo · Jessica Schrouff · Alexander D'Amour -
2023 Workshop: The 2nd Workshop on New Frontiers in Adversarial Machine Learning »
Sijia Liu · Pin-Yu Chen · Dongxiao Zhu · Eric Wong · Kathrin Grosse · Baharan Mirzasoleiman · Sanmi Koyejo -
2022 Workshop: New Frontiers in Adversarial Machine Learning »
Sijia Liu · Pin-Yu Chen · Dongxiao Zhu · Eric Wong · Kathrin Grosse · Hima Lakkaraju · Oluwasanmi Koyejo -
2022 Poster: Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularization »
Xiaojun Xu · Yibo Zhang · Evelyn Ma · Hyun Ho Son · Oluwasanmi Koyejo · Bo Li -
2022 Spotlight: Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularization »
Xiaojun Xu · Yibo Zhang · Evelyn Ma · Hyun Ho Son · Oluwasanmi Koyejo · Bo Li -
2021 Poster: Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability »
Kaizhao Liang · Yibo Zhang · Boxin Wang · Zhuolin Yang · Oluwasanmi Koyejo · Bo Li -
2021 Spotlight: Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability »
Kaizhao Liang · Yibo Zhang · Boxin Wang · Zhuolin Yang · Oluwasanmi Koyejo · Bo Li -
2021 Poster: Optimizing Black-box Metrics with Iterative Example Weighting »
Gaurush Hiranandani · Jatin Mathur · Harikrishna Narasimhan · Mahdi Milani Fard · Oluwasanmi Koyejo -
2021 Spotlight: Optimizing Black-box Metrics with Iterative Example Weighting »
Gaurush Hiranandani · Jatin Mathur · Harikrishna Narasimhan · Mahdi Milani Fard · Oluwasanmi Koyejo -
2020 Poster: On the consistency of top-k surrogate losses »
Forest Yang · Oluwasanmi Koyejo -
2020 Poster: Optimization and Analysis of the pAp@k Metric for Recommender Systems »
Gaurush Hiranandani · Warut Vijitbenjaronk · Oluwasanmi Koyejo · Prateek Jain -
2019 Poster: Partially Linear Additive Gaussian Graphical Models »
Sinong Geng · Minhao Yan · Mladen Kolar · Oluwasanmi Koyejo -
2019 Oral: Partially Linear Additive Gaussian Graphical Models »
Sinong Geng · Minhao Yan · Mladen Kolar · Oluwasanmi Koyejo -
2019 Poster: Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance »
Cong Xie · Oluwasanmi Koyejo · Indranil Gupta -
2019 Oral: Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance »
Cong Xie · Oluwasanmi Koyejo · Indranil Gupta -
2018 Poster: Binary Classification with Karmic, Threshold-Quasi-Concave Metrics »
Bowei Yan · Oluwasanmi Koyejo · Kai Zhong · Pradeep Ravikumar -
2018 Oral: Binary Classification with Karmic, Threshold-Quasi-Concave Metrics »
Bowei Yan · Oluwasanmi Koyejo · Kai Zhong · Pradeep Ravikumar -
2017 Poster: Consistency Analysis for Binary Classification Revisited »
Krzysztof Dembczynski · Wojciech Kotlowski · Oluwasanmi Koyejo · Nagarajan Natarajan -
2017 Talk: Consistency Analysis for Binary Classification Revisited »
Krzysztof Dembczynski · Wojciech Kotlowski · Oluwasanmi Koyejo · Nagarajan Natarajan