Election coding for distributed learning: Protecting SignSGD against byzantine attacks

Jy Yong Sohn, Dong Jun Han, Beongjun Choi, Jaekyun Moon

Research output: Contribution to journalConference articlepeer-review


Current distributed learning systems suffer from serious performance degradation under Byzantine attacks. This paper proposes ELECTION CODING, a coding-theoretic framework to guarantee Byzantine-robustness for distributed learning algorithms based on signed stochastic gradient descent (SignSGD) that minimizes the worker-master communication load. The suggested framework explores new information-theoretic limits of finding the majority opinion when some workers could be attacked by adversary, and paves the road to implement robust and communication-efficient distributed learning algorithms. Under this framework, we construct two types of codes, random Bernoulli codes and deterministic algebraic codes, that tolerate Byzantine attacks with a controlled amount of computational redundancy and guarantee convergence in general non-convex scenarios. For the Bernoulli codes, we provide an upper bound on the error probability in estimating the signs of the true gradients, which gives useful insights into code design for Byzantine tolerance. The proposed deterministic codes are proven to perfectly tolerate arbitrary Byzantine attacks. Experiments on real datasets confirm that the suggested codes provide substantial improvement in Byzantine tolerance of distributed learning systems employing SignSGD.

Original languageEnglish
JournalAdvances in Neural Information Processing Systems
Publication statusPublished - 2020
Event34th Conference on Neural Information Processing Systems, NeurIPS 2020 - Virtual, Online
Duration: 2020 Dec 62020 Dec 12

Bibliographical note

Funding Information:
This work was supported by IITP funds from MSIT of Korea (No. 2016-0-00563 and No. 2020-0-00626) and by National Research Foundation of Korea (No. 2019R1I1A2A02061135 and No. 2020R1A6A3A03039520).

Publisher Copyright:
© 2020 Neural information processing systems foundation. All rights reserved.

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing


Dive into the research topics of 'Election coding for distributed learning: Protecting SignSGD against byzantine attacks'. Together they form a unique fingerprint.

Cite this