Timezone: »

 
Poster
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients
Lingjiao Chen · Hongyi Wang · Zachary Charles · Dimitris Papailiopoulos

Fri Jul 13 09:15 AM -- 12:00 PM (PDT) @ Hall B #125
in Posters Fri »

Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i.e., nodes that use malicious updates to corrupt the global model stored at a parameter server (PS). To guarantee some form of robustness, recent work suggests using variants of the geometric median as an aggregation rule, in place of gradient averaging. Unfortunately, median-based rules can incur a prohibitive computational overhead in large-scale settings, and their convergence guarantees often require strong assumptions. In this work, we present DRACO, a scalable framework for robust distributed training that uses ideas from coding theory. In DRACO, each compute node evaluates redundant gradients that are used by the parameter server to eliminate the effects of adversarial updates. DRACO comes with problem-independent robustness guarantees, and the model that it trains is identical to the one trained in the adversary-free setup. We provide extensive experiments on real datasets and distributed setups across a variety of large-scale models, where we show that DRACO is several times, to orders of magnitude faster than median-based approaches.

[ PDF] 

Author Information

Lingjiao Chen (University of Wisconsin-Madison)
Hongyi Wang (University of Wisconsin-Madison)

I’m currently a second-year Ph.D. student at Computer Sciences Department of University of Wisconsin - Madison, advised by Prof. Dimitris Papailiopoulos. My research interests locate in machine learning, distributed system, and large-scale optimization.

Zachary Charles (University of Wisconsin-Madison)
Dimitris Papailiopoulos (ECE at University of Wisconsin-Madison)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors