Local Critic Training for Model-Parallel Learning of Deep Neural Networks

Hojung Lee, Cho Jui Hsieh, Jong Seok Lee

Research output: Contribution to journalArticlepeer-review

4 Citations (Scopus)


In this article, we propose a novel model-parallel learning method, called local critic training, which trains neural networks using additional modules called local critic networks. The main network is divided into several layer groups, and each layer group is updated through error gradients estimated by the corresponding local critic network. We show that the proposed approach successfully decouples the update process of the layer groups for both convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In addition, we demonstrate that the proposed method is guaranteed to converge to a critical point. We also show that trained networks by the proposed method can be used for structural optimization. Experimental results show that our method achieves satisfactory performance, reduces training time greatly, and decreases memory consumption per machine. Code is available at https://github.com/hjdw2/Local-critic-training.

Original languageEnglish
Pages (from-to)4424-4436
Number of pages13
JournalIEEE Transactions on Neural Networks and Learning Systems
Issue number9
Publication statusPublished - 2022 Sept 1

Bibliographical note

Publisher Copyright:
© 2012 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence


Dive into the research topics of 'Local Critic Training for Model-Parallel Learning of Deep Neural Networks'. Together they form a unique fingerprint.

Cite this