Emerging mobile services heavily utilize Neural Networks (NNs) to improve user experiences. Such NN-assisted services depend on fast NN execution for high responsiveness, demanding mobile devices to minimize the NN execution latency by efficiently utilizing their underlying hardware resources. To better utilize the resources, existing mobile NN frameworks either employ various CPU-friendly optimizations (e.g., vectorization, quantization) or exploit data parallelism using heterogeneous processors such as GPUs and DSPs. However, their performance is still bounded by the performance of the single target processor, so that real-time services such as voice-driven search often fail to react to user requests in time. It is obvious that this problem will become more serious with the introduction of more demanding NN-assisted services. In this paper, we propose µLayer, a low latency on-device inference runtime which significantly improves the latency of NN-assisted services. µLayer accelerates each NN layer by simultaneously utilizing diverse heterogeneous processors on a mobile device and by performing computations using processor-friendly quantization. Two key findings motivate our work: 1) the existing frameworks are limited by single-processor performance as they execute an NN layer using only a single processor, and 2) the CPU and the GPU on the same mobile device achieve comparable computational throughput, making cooperative acceleration highly promising. First, to accelerate an NN layer using both the CPU and the GPU at the same time, µLayer employs a layer distribution mechanism which completely removes redundant computations between the processors. Next, µLayer optimizes the per-processor performance by making the processors utilize different data types that maximize their utilization. In addition, to minimize potential latency increases due to overly aggressive workload distribution, µLayer selectively increases the distribution granularity to divergent layer paths. Our experiments using representative NNs and mobile devices show that µLayer significantly improves the speed and the energy efficiency of on-device inference by up to 69.6% and 58.1%, respectively, over the state-of-the-art NN execution mechanism.
|Title of host publication||Proceedings of the 14th EuroSys Conference 2019|
|Publisher||Association for Computing Machinery, Inc|
|Publication status||Published - 2019 Mar 25|
|Event||14th European Conference on Computer Systems, EuroSys 2019 - Dresden, Germany|
Duration: 2019 Mar 25 → 2019 Mar 28
|Name||Proceedings of the 14th EuroSys Conference 2019|
|Conference||14th European Conference on Computer Systems, EuroSys 2019|
|Period||19/3/25 → 19/3/28|
Bibliographical noteFunding Information:
This work was partly supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (MSIT) (NRF-2015M3C4A7065647, NRF-2017R1A2B3011038), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.1711080972, Neuromorphic Computing Software Platform for Artificial Intelligence Systems), and Creative Pioneering Researchers Program through Seoul National University. We also appreciate the support from Automation and Systems Research Institute (ASRI), Inter-university Semiconductor Research Center (ISRC), and Neural Processing Research Center (NPRC) at Seoul National University.
All Science Journal Classification (ASJC) codes
- Hardware and Architecture
- Electrical and Electronic Engineering