Learning to tell brake and turn signals in videos using CNN-LSTM structure

Han Kai Hsu; Yi Hsuan Tsai; Xue Mei; Kuan Hui Lee; Naoki Nagasaka; Danil Prokhorov; Ming Hsuan Yang

doi:10.1109/ITSC.2017.8317782

Learning to tell brake and turn signals in videos using CNN-LSTM structure

Han Kai Hsu, Yi Hsuan Tsai, Xue Mei, Kuan Hui Lee, Naoki Nagasaka, Danil Prokhorov, Ming Hsuan Yang

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

9 Citations (Scopus)

Abstract

We present a method that learns to tell rear signals from a number of frames using a deep learning framework. The proposed framework extracts spatial features with a convolution neural network (CNN), and then applies a long short term memory (LSTM) network to learn the long-term dependencies. The brake signal classifier is trained using RGB frames, while the turn signal is recognized via a two-step localization approach. The two separate classifiers are learned to recognize the static brake signals and the dynamic turn signals. As a result, our recognition system can recognize 8 different rear signals via the combined two classifiers in real-world traffic scenes. Experimental results show that our method is able to obtain more accurate predictions than using only the CNN to classify rear signals with time sequence inputs.

Original language	English
Title of host publication	2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1-6
Number of pages	6
ISBN (Electronic)	9781538615256
DOIs	https://doi.org/10.1109/ITSC.2017.8317782
Publication status	Published - 2018 Mar 14
Event	20th IEEE International Conference on Intelligent Transportation Systems, ITSC 2017 - Yokohama, Kanagawa, Japan Duration: 2017 Oct 16 → 2017 Oct 19

Publication series

Name	IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC
Volume	2018-March

Conference

Conference	20th IEEE International Conference on Intelligent Transportation Systems, ITSC 2017
Country/Territory	Japan
City	Yokohama, Kanagawa
Period	17/10/16 → 17/10/19

Bibliographical note

Publisher Copyright:
© 2017 IEEE.

All Science Journal Classification (ASJC) codes

Automotive Engineering
Mechanical Engineering
Computer Science Applications

Access to Document

10.1109/ITSC.2017.8317782

Cite this

Hsu, H. K., Tsai, Y. H., Mei, X., Lee, K. H., Nagasaka, N., Prokhorov, D., & Yang, M. H. (2018). Learning to tell brake and turn signals in videos using CNN-LSTM structure. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017 (pp. 1-6). (IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC; Vol. 2018-March). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ITSC.2017.8317782

@inproceedings{9402b80a2e8b4190abbf8e5ab4db6925,

title = "Learning to tell brake and turn signals in videos using CNN-LSTM structure",

abstract = "We present a method that learns to tell rear signals from a number of frames using a deep learning framework. The proposed framework extracts spatial features with a convolution neural network (CNN), and then applies a long short term memory (LSTM) network to learn the long-term dependencies. The brake signal classifier is trained using RGB frames, while the turn signal is recognized via a two-step localization approach. The two separate classifiers are learned to recognize the static brake signals and the dynamic turn signals. As a result, our recognition system can recognize 8 different rear signals via the combined two classifiers in real-world traffic scenes. Experimental results show that our method is able to obtain more accurate predictions than using only the CNN to classify rear signals with time sequence inputs.",

author = "Hsu, {Han Kai} and Tsai, {Yi Hsuan} and Xue Mei and Lee, {Kuan Hui} and Naoki Nagasaka and Danil Prokhorov and Yang, {Ming Hsuan}",

note = "Publisher Copyright: {\textcopyright} 2017 IEEE.; 20th IEEE International Conference on Intelligent Transportation Systems, ITSC 2017 ; Conference date: 16-10-2017 Through 19-10-2017",

year = "2018",

month = mar,

day = "14",

doi = "10.1109/ITSC.2017.8317782",

language = "English",

series = "IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1--6",

booktitle = "2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017",

address = "United States",

}

Hsu, HK, Tsai, YH, Mei, X, Lee, KH, Nagasaka, N, Prokhorov, D & Yang, MH 2018, Learning to tell brake and turn signals in videos using CNN-LSTM structure. in 2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017. IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-March, Institute of Electrical and Electronics Engineers Inc., pp. 1-6, 20th IEEE International Conference on Intelligent Transportation Systems, ITSC 2017, Yokohama, Kanagawa, Japan, 17/10/16. https://doi.org/10.1109/ITSC.2017.8317782

Learning to tell brake and turn signals in videos using CNN-LSTM structure. / Hsu, Han Kai; Tsai, Yi Hsuan; Mei, Xue et al.

2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1-6 (IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC; Vol. 2018-March).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Learning to tell brake and turn signals in videos using CNN-LSTM structure

AU - Hsu, Han Kai

AU - Tsai, Yi Hsuan

AU - Mei, Xue

AU - Lee, Kuan Hui

AU - Nagasaka, Naoki

AU - Prokhorov, Danil

AU - Yang, Ming Hsuan

PY - 2018/3/14

Y1 - 2018/3/14

N2 - We present a method that learns to tell rear signals from a number of frames using a deep learning framework. The proposed framework extracts spatial features with a convolution neural network (CNN), and then applies a long short term memory (LSTM) network to learn the long-term dependencies. The brake signal classifier is trained using RGB frames, while the turn signal is recognized via a two-step localization approach. The two separate classifiers are learned to recognize the static brake signals and the dynamic turn signals. As a result, our recognition system can recognize 8 different rear signals via the combined two classifiers in real-world traffic scenes. Experimental results show that our method is able to obtain more accurate predictions than using only the CNN to classify rear signals with time sequence inputs.

AB - We present a method that learns to tell rear signals from a number of frames using a deep learning framework. The proposed framework extracts spatial features with a convolution neural network (CNN), and then applies a long short term memory (LSTM) network to learn the long-term dependencies. The brake signal classifier is trained using RGB frames, while the turn signal is recognized via a two-step localization approach. The two separate classifiers are learned to recognize the static brake signals and the dynamic turn signals. As a result, our recognition system can recognize 8 different rear signals via the combined two classifiers in real-world traffic scenes. Experimental results show that our method is able to obtain more accurate predictions than using only the CNN to classify rear signals with time sequence inputs.

UR - http://www.scopus.com/inward/record.url?scp=85046268514&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85046268514&partnerID=8YFLogxK

U2 - 10.1109/ITSC.2017.8317782

DO - 10.1109/ITSC.2017.8317782

M3 - Conference contribution

AN - SCOPUS:85046268514

T3 - IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC

SP - 1

EP - 6

BT - 2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 20th IEEE International Conference on Intelligent Transportation Systems, ITSC 2017

Y2 - 16 October 2017 through 19 October 2017

ER -

Hsu HK, Tsai YH, Mei X, Lee KH, Nagasaka N, Prokhorov D et al. Learning to tell brake and turn signals in videos using CNN-LSTM structure. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1-6. (IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC). doi: 10.1109/ITSC.2017.8317782

Learning to tell brake and turn signals in videos using CNN-LSTM structure

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this