Timezone: »

 
Poster
Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning
Dong Yin · Yudong Chen · Kannan Ramchandran · Peter Bartlett

Thu Jun 13 06:30 PM -- 09:00 PM (PDT) @ Pacific Ballroom #91
in Posters Thu »

We study robust distributed learning that involves minimizing a non-convex loss function with saddle points. We consider the Byzantine setting where some worker machines have abnormal or even arbitrary and adversarial behavior, and in this setting, the Byzantine machines may create fake local minima near a saddle point that is far away from any true local minimum, even when robust gradient estimators are used. We develop ByzantinePGD, a robust first-order algorithm that can provably escape saddle points and fake local minima, and converge to an approximate true local minimizer with low iteration complexity. As a by-product, we give a simpler algorithm and analysis for escaping saddle points in the usual non-Byzantine setting. We further discuss three robust gradient estimators that can be used in ByzantinePGD, including median, trimmed mean, and iterative filtering. We characterize their performance in concrete statistical settings, and argue for their near-optimality in low and high dimensional regimes.

Keywords: Non-convex Optimization · Parallel and Distributed Learning · Robust Statistics and Machine Learning

Author Information

Dong Yin (UC Berkeley)
Yudong Chen (Cornell University)
Kannan Ramchandran (UC Berkeley)
Peter Bartlett (UC Berkeley)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors