Learning video-story composition via recurrent neural network

Guangyu Zhong, Yi Hsuan Tsai, Sifei Liu, Zhixun Su, Ming Hsuan Yang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatialoral semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1727-1735
Number of pages9
ISBN (Electronic)9781538648865
DOIs
Publication statusPublished - 2018 May 3
Event18th IEEE Winter Conference on Applications of Computer Vision, WACV 2018 - Lake Tahoe, United States
Duration: 2018 Mar 122018 Mar 15

Publication series

NameProceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018
Volume2018-January

Conference

Conference18th IEEE Winter Conference on Applications of Computer Vision, WACV 2018
CountryUnited States
CityLake Tahoe
Period18/3/1218/3/15

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Computer Vision and Pattern Recognition

Cite this

Zhong, G., Tsai, Y. H., Liu, S., Su, Z., & Yang, M. H. (2018). Learning video-story composition via recurrent neural network. In Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018 (pp. 1727-1735). (Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018; Vol. 2018-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/WACV.2018.00192