Learning video-story composition via recurrent neural network

Guangyu Zhong; Yi Hsuan Tsai; Sifei Liu; Zhixun Su; Ming Hsuan Yang

doi:10.1109/WACV.2018.00192

Learning video-story composition via recurrent neural network

Guangyu Zhong, Yi Hsuan Tsai, Sifei Liu, Zhixun Su, Ming Hsuan Yang

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatialoral semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.

Original language	English
Title of host publication	Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1727-1735
Number of pages	9
ISBN (Electronic)	9781538648865
DOIs	https://doi.org/10.1109/WACV.2018.00192
Publication status	Published - 2018 May 3
Event	18th IEEE Winter Conference on Applications of Computer Vision, WACV 2018 - Lake Tahoe, United States Duration: 2018 Mar 12 → 2018 Mar 15

Publication series

Name	Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018
Volume	2018-January

Conference

Conference	18th IEEE Winter Conference on Applications of Computer Vision, WACV 2018
Country	United States
City	Lake Tahoe
Period	18/3/12 → 18/3/15

Bibliographical note

Funding Information:
Acknowledgements: This work is supported in part by NS-FC (No. 61572099 and 61522203), NSF CAREER (No. 1149783), 973 Program (No. 2014CB347600), NSF of Jiangsu Province (No. BK20140058), the National Key R&D Program of China (No. 2016YFB1001001), and gifts from Adobe and Nvidia.

All Science Journal Classification (ASJC) codes

Computer Science Applications
Computer Vision and Pattern Recognition

Access to Document

10.1109/WACV.2018.00192

Fingerprint Dive into the research topics of 'Learning video-story composition via recurrent neural network'. Together they form a unique fingerprint.

Cite this

Zhong, G., Tsai, Y. H., Liu, S., Su, Z., & Yang, M. H. (2018). Learning video-story composition via recurrent neural network. In Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018 (pp. 1727-1735). (Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018; Vol. 2018-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/WACV.2018.00192

Zhong, Guangyu ; Tsai, Yi Hsuan ; Liu, Sifei ; Su, Zhixun ; Yang, Ming Hsuan. / Learning video-story composition via recurrent neural network. Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1727-1735 (Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018).

@inproceedings{8c763cafb9df42bf920127f845bc5326,

title = "Learning video-story composition via recurrent neural network",

abstract = "In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatialoral semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.",

author = "Guangyu Zhong and Tsai, {Yi Hsuan} and Sifei Liu and Zhixun Su and Yang, {Ming Hsuan}",

note = "Funding Information: Acknowledgements: This work is supported in part by NS-FC (No. 61572099 and 61522203), NSF CAREER (No. 1149783), 973 Program (No. 2014CB347600), NSF of Jiangsu Province (No. BK20140058), the National Key R&D Program of China (No. 2016YFB1001001), and gifts from Adobe and Nvidia.; 18th IEEE Winter Conference on Applications of Computer Vision, WACV 2018 ; Conference date: 12-03-2018 Through 15-03-2018",

year = "2018",

month = may,

day = "3",

doi = "10.1109/WACV.2018.00192",

language = "English",

series = "Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1727--1735",

booktitle = "Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018",

address = "United States",

}

Zhong, G, Tsai, YH, Liu, S, Su, Z & Yang, MH 2018, Learning video-story composition via recurrent neural network. in Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018. Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, vol. 2018-January, Institute of Electrical and Electronics Engineers Inc., pp. 1727-1735, 18th IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, United States, 18/3/12. https://doi.org/10.1109/WACV.2018.00192

Learning video-story composition via recurrent neural network. / Zhong, Guangyu; Tsai, Yi Hsuan; Liu, Sifei; Su, Zhixun; Yang, Ming Hsuan.

Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1727-1735 (Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018; Vol. 2018-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Learning video-story composition via recurrent neural network

AU - Zhong, Guangyu

AU - Tsai, Yi Hsuan

AU - Liu, Sifei

AU - Su, Zhixun

AU - Yang, Ming Hsuan

N1 - Funding Information: Acknowledgements: This work is supported in part by NS-FC (No. 61572099 and 61522203), NSF CAREER (No. 1149783), 973 Program (No. 2014CB347600), NSF of Jiangsu Province (No. BK20140058), the National Key R&D Program of China (No. 2016YFB1001001), and gifts from Adobe and Nvidia.

PY - 2018/5/3

Y1 - 2018/5/3

N2 - In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatialoral semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.

AB - In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatialoral semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.

UR - http://www.scopus.com/inward/record.url?scp=85050962841&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050962841&partnerID=8YFLogxK

U2 - 10.1109/WACV.2018.00192

DO - 10.1109/WACV.2018.00192

M3 - Conference contribution

AN - SCOPUS:85050962841

T3 - Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018

SP - 1727

EP - 1735

BT - Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 18th IEEE Winter Conference on Applications of Computer Vision, WACV 2018

Y2 - 12 March 2018 through 15 March 2018

ER -

Zhong G, Tsai YH, Liu S, Su Z, Yang MH. Learning video-story composition via recurrent neural network. In Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1727-1735. (Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018). https://doi.org/10.1109/WACV.2018.00192