Legion: Tailoring Grouped Neural Execution Considering Heterogeneity on Multiple Edge Devices

Kyunghwan Choi, Seongju Lee, Beom Woo Kang, Yongjun Park

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Distributing workloads that cannot be handled by a single edge device across multiple edge devices is a promising solution that minimizes the inference latency of deep learning applications by exploiting model parallelism. Several prior solutions have been proposed to partition target models efficiently, but most studies have focused on finding the optimal fused layer configurations, which minimize the data-transfer overhead between layers. However, as recent deep learning network models have become more complex and the ability to deploy them quickly has become a key challenge, the search for the best fused layer configurations of target models has become a major requirement. To solve this problem, we propose a lightweight model partitioning framework called Legion to find the optimal fused layer configurations with minimal profiling execution trials. By finding the optimal configurations using cost matrix construction and wild card selection, the experimental results showed that Legion achieved a similar performance to the full configuration search at a fraction of the search time. Moreover, Legion performed effectively even on a group of heterogeneous target devices by introducing a per-device cost-related matrix construction. With three popular networks, Legion shows only 3.4% performance loss as compared to a full searching scheme (FSS), on various different device configurations consisting of up to six heterogeneous devices, and minimizes the profiling overhead by 48.7× on average.

Original languageEnglish
Title of host publicationProceedings - 2021 IEEE 39th International Conference on Computer Design, ICCD 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages383-390
Number of pages8
ISBN (Electronic)9781665432191
DOIs
Publication statusPublished - 2021
Event39th IEEE International Conference on Computer Design, ICCD 2021 - Virtual, Online, United States
Duration: 2021 Oct 242021 Oct 27

Publication series

NameProceedings - IEEE International Conference on Computer Design: VLSI in Computers and Processors
Volume2021-October
ISSN (Print)1063-6404

Conference

Conference39th IEEE International Conference on Computer Design, ICCD 2021
Country/TerritoryUnited States
CityVirtual, Online
Period21/10/2421/10/27

Bibliographical note

Funding Information:
ACKNOWLEDGMENT This work was supported by Samsung Research Funding & Incubation Center of Samsung Electronics under Project Number SRFC-IT1901-03, and by the research fund of Hanyang University (HY-2017). Yongjun Park is the corresponding author.

Publisher Copyright:
© 2021 IEEE.

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Legion: Tailoring Grouped Neural Execution Considering Heterogeneity on Multiple Edge Devices'. Together they form a unique fingerprint.

Cite this