Skip to yearly menu bar Skip to main content


Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning

Dong Yin · Yudong Chen · Kannan Ramchandran · Peter Bartlett

Pacific Ballroom #91

Keywords: [ Robust Statistics and Machine Learning ] [ Parallel and Distributed Learning ] [ Non-convex Optimization ]


We study robust distributed learning that involves minimizing a non-convex loss function with saddle points. We consider the Byzantine setting where some worker machines have abnormal or even arbitrary and adversarial behavior, and in this setting, the Byzantine machines may create fake local minima near a saddle point that is far away from any true local minimum, even when robust gradient estimators are used. We develop ByzantinePGD, a robust first-order algorithm that can provably escape saddle points and fake local minima, and converge to an approximate true local minimizer with low iteration complexity. As a by-product, we give a simpler algorithm and analysis for escaping saddle points in the usual non-Byzantine setting. We further discuss three robust gradient estimators that can be used in ByzantinePGD, including median, trimmed mean, and iterative filtering. We characterize their performance in concrete statistical settings, and argue for their near-optimality in low and high dimensional regimes.

Live content is unavailable. Log in and register to view live content