Learning to tell brake and turn signals in videos using CNN-LSTM structure

Han Kai Hsu, Yi Hsuan Tsai, Xue Mei, Kuan Hui Lee, Naoki Nagasaka, Danil Prokhorov, Ming Hsuan Yang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a method that learns to tell rear signals from a number of frames using a deep learning framework. The proposed framework extracts spatial features with a convolution neural network (CNN), and then applies a long short term memory (LSTM) network to learn the long-term dependencies. The brake signal classifier is trained using RGB frames, while the turn signal is recognized via a two-step localization approach. The two separate classifiers are learned to recognize the static brake signals and the dynamic turn signals. As a result, our recognition system can recognize 8 different rear signals via the combined two classifiers in real-world traffic scenes. Experimental results show that our method is able to obtain more accurate predictions than using only the CNN to classify rear signals with time sequence inputs.

Original languageEnglish
Title of host publication2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-6
Number of pages6
ISBN (Electronic)9781538615256
DOIs
Publication statusPublished - 2018 Mar 14
Event20th IEEE International Conference on Intelligent Transportation Systems, ITSC 2017 - Yokohama, Kanagawa, Japan
Duration: 2017 Oct 162017 Oct 19

Publication series

NameIEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC
Volume2018-March

Conference

Conference20th IEEE International Conference on Intelligent Transportation Systems, ITSC 2017
CountryJapan
CityYokohama, Kanagawa
Period17/10/1617/10/19

Fingerprint

Convolution
Brakes
Classifiers
Neural networks
Long short-term memory

All Science Journal Classification (ASJC) codes

  • Automotive Engineering
  • Mechanical Engineering
  • Computer Science Applications

Cite this

Hsu, H. K., Tsai, Y. H., Mei, X., Lee, K. H., Nagasaka, N., Prokhorov, D., & Yang, M. H. (2018). Learning to tell brake and turn signals in videos using CNN-LSTM structure. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017 (pp. 1-6). (IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC; Vol. 2018-March). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ITSC.2017.8317782
Hsu, Han Kai ; Tsai, Yi Hsuan ; Mei, Xue ; Lee, Kuan Hui ; Nagasaka, Naoki ; Prokhorov, Danil ; Yang, Ming Hsuan. / Learning to tell brake and turn signals in videos using CNN-LSTM structure. 2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1-6 (IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC).
@inproceedings{9402b80a2e8b4190abbf8e5ab4db6925,
title = "Learning to tell brake and turn signals in videos using CNN-LSTM structure",
abstract = "We present a method that learns to tell rear signals from a number of frames using a deep learning framework. The proposed framework extracts spatial features with a convolution neural network (CNN), and then applies a long short term memory (LSTM) network to learn the long-term dependencies. The brake signal classifier is trained using RGB frames, while the turn signal is recognized via a two-step localization approach. The two separate classifiers are learned to recognize the static brake signals and the dynamic turn signals. As a result, our recognition system can recognize 8 different rear signals via the combined two classifiers in real-world traffic scenes. Experimental results show that our method is able to obtain more accurate predictions than using only the CNN to classify rear signals with time sequence inputs.",
author = "Hsu, {Han Kai} and Tsai, {Yi Hsuan} and Xue Mei and Lee, {Kuan Hui} and Naoki Nagasaka and Danil Prokhorov and Yang, {Ming Hsuan}",
year = "2018",
month = "3",
day = "14",
doi = "10.1109/ITSC.2017.8317782",
language = "English",
series = "IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "1--6",
booktitle = "2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017",
address = "United States",

}

Hsu, HK, Tsai, YH, Mei, X, Lee, KH, Nagasaka, N, Prokhorov, D & Yang, MH 2018, Learning to tell brake and turn signals in videos using CNN-LSTM structure. in 2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017. IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-March, Institute of Electrical and Electronics Engineers Inc., pp. 1-6, 20th IEEE International Conference on Intelligent Transportation Systems, ITSC 2017, Yokohama, Kanagawa, Japan, 17/10/16. https://doi.org/10.1109/ITSC.2017.8317782

Learning to tell brake and turn signals in videos using CNN-LSTM structure. / Hsu, Han Kai; Tsai, Yi Hsuan; Mei, Xue; Lee, Kuan Hui; Nagasaka, Naoki; Prokhorov, Danil; Yang, Ming Hsuan.

2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1-6 (IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC; Vol. 2018-March).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Learning to tell brake and turn signals in videos using CNN-LSTM structure

AU - Hsu, Han Kai

AU - Tsai, Yi Hsuan

AU - Mei, Xue

AU - Lee, Kuan Hui

AU - Nagasaka, Naoki

AU - Prokhorov, Danil

AU - Yang, Ming Hsuan

PY - 2018/3/14

Y1 - 2018/3/14

N2 - We present a method that learns to tell rear signals from a number of frames using a deep learning framework. The proposed framework extracts spatial features with a convolution neural network (CNN), and then applies a long short term memory (LSTM) network to learn the long-term dependencies. The brake signal classifier is trained using RGB frames, while the turn signal is recognized via a two-step localization approach. The two separate classifiers are learned to recognize the static brake signals and the dynamic turn signals. As a result, our recognition system can recognize 8 different rear signals via the combined two classifiers in real-world traffic scenes. Experimental results show that our method is able to obtain more accurate predictions than using only the CNN to classify rear signals with time sequence inputs.

AB - We present a method that learns to tell rear signals from a number of frames using a deep learning framework. The proposed framework extracts spatial features with a convolution neural network (CNN), and then applies a long short term memory (LSTM) network to learn the long-term dependencies. The brake signal classifier is trained using RGB frames, while the turn signal is recognized via a two-step localization approach. The two separate classifiers are learned to recognize the static brake signals and the dynamic turn signals. As a result, our recognition system can recognize 8 different rear signals via the combined two classifiers in real-world traffic scenes. Experimental results show that our method is able to obtain more accurate predictions than using only the CNN to classify rear signals with time sequence inputs.

UR - http://www.scopus.com/inward/record.url?scp=85046268514&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85046268514&partnerID=8YFLogxK

U2 - 10.1109/ITSC.2017.8317782

DO - 10.1109/ITSC.2017.8317782

M3 - Conference contribution

AN - SCOPUS:85046268514

T3 - IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC

SP - 1

EP - 6

BT - 2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Hsu HK, Tsai YH, Mei X, Lee KH, Nagasaka N, Prokhorov D et al. Learning to tell brake and turn signals in videos using CNN-LSTM structure. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems, ITSC 2017. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1-6. (IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC). https://doi.org/10.1109/ITSC.2017.8317782