Real-Time Object Detection System with Multi-Path Neural Networks

Seonyeong Heo, Sungjun Cho, Youngsok Kim, Hanjun Kim

Research output: Chapter in Book/Report/Conference proceedingConference contribution

29 Citations (Scopus)


Thanks to the recent advances in Deep Neural Networks (DNNs), DNN-based object detection systems become highly accurate and widely used in real-time environments such as autonomous vehicles, drones and security robots. Although the systems should detect objects within a certain time limit that can vary depending on their execution environments such as vehicle speeds, existing systems blindly execute the entire long-latency DNNs without reflecting the time-varying time limits, and thus they cannot guarantee real-time constraints. This work proposes a novel real-time object detection system that employs multipath neural networks based on a new worst-case execution time (WCET) model for DNNs on a GPU. This work designs the WCET model for a single DNN layer analyzing processor and memory contention on GPUs, and extends the WCET model to the end-to-end networks. This work also designs the multipath networks with three new operators such as skip, switch, and dynamic generate proposals that dynamically change their execution paths and the number of target objects. Finally, this work proposes a path decision model that chooses the optimal execution path at run-time reflecting dynamically changing environments and time constraints. Our detailed evaluation using widely-used driving datasets shows that the proposed real-time object detection system performs as good as a baseline object detection system without violating the time-varying time limits. Moreover, the WCET model predicts the worst-case execution latency of convolutional and group normalization layers with only 27% and 81% errors on average, respectively.

Original languageEnglish
Title of host publicationProceedings - 2020 IEEE Real-Time and Embedded Technology and Applications Symposium, RTAS 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages14
ISBN (Electronic)9781728154992
Publication statusPublished - 2020 Apr
Event26th IEEE Real-Time and Embedded Technology and Applications Symposium, RTAS 2020 - Sydney, Australia
Duration: 2020 Apr 212020 Apr 24

Publication series

NameProceedings of the IEEE Real-Time and Embedded Technology and Applications Symposium, RTAS
ISSN (Print)1545-3421


Conference26th IEEE Real-Time and Embedded Technology and Applications Symposium, RTAS 2020

Bibliographical note

Funding Information:
We thank the anonymous reviewers and shepherd for their valuable feedback. This research was supported by Sam-sung Research Funding Center of Samsung Electronics under Project Number SRFC-TB1703-03. This research was also partly supported by the Yonsei university research fund of 2019. Hanjun Kim is the corresponding author of this paper.

Publisher Copyright:
© 2020 IEEE.

All Science Journal Classification (ASJC) codes

  • Engineering(all)


Dive into the research topics of 'Real-Time Object Detection System with Multi-Path Neural Networks'. Together they form a unique fingerprint.

Cite this