Character Grounding and Re-identification in Story of Videos and Text Descriptions

Youngjae Yu; Jongseok Kim; Heeseung Yun; Jiwan Chung; Gunhee Kim

doi:10.1007/978-3-030-58558-7_32

Character Grounding and Re-identification in Story of Videos and Text Descriptions

Youngjae Yu, Jongseok Kim, Heeseung Yun, Jiwan Chung, Gunhee Kim

Department of Artificial Intelligence

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We address character grounding and re-identification in multiple story-based videos like movies and associated text descriptions. In order to solve these related tasks in a mutually rewarding way, we propose a model named Character in Story Identification Network (CiSIN). Our method builds two semantically informative representations via joint training of multiple objectives for character grounding, video/text re-identification and gender prediction: Visual Track Embedding from videos and Textual Character Embedding from text context. These two representations are learned to retain rich semantic multimodal information that enables even simple MLPs to achieve the state-of-the-art performance on the target tasks. More specifically, our CiSIN model achieves the best performance in the Fill-in the Characters task of LSMDC 2019 challenges[35]. Moreover, it outperforms previous state-of-the-art models in M-VAD Names dataset [30] as a benchmark of multimodal character grounding and re-identification.

Original language	English
Title of host publication	Computer Vision – ECCV 2020 - 16th European Conference, Proceedings
Editors	Andrea Vedaldi, Horst Bischof, Thomas Brox, Jan-Michael Frahm
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	543-559
Number of pages	17
ISBN (Print)	9783030585570
DOIs	https://doi.org/10.1007/978-3-030-58558-7_32
Publication status	Published - 2020
Event	16th European Conference on Computer Vision, ECCV 2020 - Glasgow, United Kingdom Duration: 2020 Aug 23 → 2020 Aug 28

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12350 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	16th European Conference on Computer Vision, ECCV 2020
Country/Territory	United Kingdom
City	Glasgow
Period	20/8/23 → 20/8/28

Bibliographical note

Funding Information:
Acknowledgement. We thank SNUVL lab members for helpful comments. This research was supported by Seoul National University, Brain Research Program by National Research Foundation of Korea (NRF) (2017M3C7A1047860), and AIR Lab (AI Research Lab) in Hyundai Motor Company through HMC-SNU AI Consortium Fund.

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-030-58558-7_32

Cite this

Yu, Y., Kim, J., Yun, H., Chung, J., & Kim, G. (2020). Character Grounding and Re-identification in Story of Videos and Text Descriptions. In A. Vedaldi, H. Bischof, T. Brox, & J-M. Frahm (Eds.), Computer Vision – ECCV 2020 - 16th European Conference, Proceedings (pp. 543-559). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12350 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-58558-7_32

Yu, Youngjae ; Kim, Jongseok ; Yun, Heeseung et al. / Character Grounding and Re-identification in Story of Videos and Text Descriptions. Computer Vision – ECCV 2020 - 16th European Conference, Proceedings. editor / Andrea Vedaldi ; Horst Bischof ; Thomas Brox ; Jan-Michael Frahm. Springer Science and Business Media Deutschland GmbH, 2020. pp. 543-559 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{a7c04d2aae1d413bab19c49d1155af2e,

title = "Character Grounding and Re-identification in Story of Videos and Text Descriptions",

abstract = "We address character grounding and re-identification in multiple story-based videos like movies and associated text descriptions. In order to solve these related tasks in a mutually rewarding way, we propose a model named Character in Story Identification Network (CiSIN). Our method builds two semantically informative representations via joint training of multiple objectives for character grounding, video/text re-identification and gender prediction: Visual Track Embedding from videos and Textual Character Embedding from text context. These two representations are learned to retain rich semantic multimodal information that enables even simple MLPs to achieve the state-of-the-art performance on the target tasks. More specifically, our CiSIN model achieves the best performance in the Fill-in the Characters task of LSMDC 2019 challenges[35]. Moreover, it outperforms previous state-of-the-art models in M-VAD Names dataset [30] as a benchmark of multimodal character grounding and re-identification.",

author = "Youngjae Yu and Jongseok Kim and Heeseung Yun and Jiwan Chung and Gunhee Kim",

note = "Funding Information: Acknowledgement. We thank SNUVL lab members for helpful comments. This research was supported by Seoul National University, Brain Research Program by National Research Foundation of Korea (NRF) (2017M3C7A1047860), and AIR Lab (AI Research Lab) in Hyundai Motor Company through HMC-SNU AI Consortium Fund. Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 16th European Conference on Computer Vision, ECCV 2020 ; Conference date: 23-08-2020 Through 28-08-2020",

year = "2020",

doi = "10.1007/978-3-030-58558-7_32",

language = "English",

isbn = "9783030585570",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "543--559",

editor = "Andrea Vedaldi and Horst Bischof and Thomas Brox and Jan-Michael Frahm",

booktitle = "Computer Vision – ECCV 2020 - 16th European Conference, Proceedings",

address = "Germany",

}

Yu, Y, Kim, J, Yun, H, Chung, J & Kim, G 2020, Character Grounding and Re-identification in Story of Videos and Text Descriptions. in A Vedaldi, H Bischof, T Brox & J-M Frahm (eds), Computer Vision – ECCV 2020 - 16th European Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12350 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 543-559, 16th European Conference on Computer Vision, ECCV 2020, Glasgow, United Kingdom, 20/8/23. https://doi.org/10.1007/978-3-030-58558-7_32

Character Grounding and Re-identification in Story of Videos and Text Descriptions. / Yu, Youngjae; Kim, Jongseok; Yun, Heeseung et al.

Computer Vision – ECCV 2020 - 16th European Conference, Proceedings. ed. / Andrea Vedaldi; Horst Bischof; Thomas Brox; Jan-Michael Frahm. Springer Science and Business Media Deutschland GmbH, 2020. p. 543-559 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12350 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Character Grounding and Re-identification in Story of Videos and Text Descriptions

AU - Yu, Youngjae

AU - Kim, Jongseok

AU - Yun, Heeseung

AU - Chung, Jiwan

AU - Kim, Gunhee

N1 - Funding Information: Acknowledgement. We thank SNUVL lab members for helpful comments. This research was supported by Seoul National University, Brain Research Program by National Research Foundation of Korea (NRF) (2017M3C7A1047860), and AIR Lab (AI Research Lab) in Hyundai Motor Company through HMC-SNU AI Consortium Fund. Publisher Copyright: © 2020, Springer Nature Switzerland AG.

PY - 2020

Y1 - 2020

N2 - We address character grounding and re-identification in multiple story-based videos like movies and associated text descriptions. In order to solve these related tasks in a mutually rewarding way, we propose a model named Character in Story Identification Network (CiSIN). Our method builds two semantically informative representations via joint training of multiple objectives for character grounding, video/text re-identification and gender prediction: Visual Track Embedding from videos and Textual Character Embedding from text context. These two representations are learned to retain rich semantic multimodal information that enables even simple MLPs to achieve the state-of-the-art performance on the target tasks. More specifically, our CiSIN model achieves the best performance in the Fill-in the Characters task of LSMDC 2019 challenges[35]. Moreover, it outperforms previous state-of-the-art models in M-VAD Names dataset [30] as a benchmark of multimodal character grounding and re-identification.

AB - We address character grounding and re-identification in multiple story-based videos like movies and associated text descriptions. In order to solve these related tasks in a mutually rewarding way, we propose a model named Character in Story Identification Network (CiSIN). Our method builds two semantically informative representations via joint training of multiple objectives for character grounding, video/text re-identification and gender prediction: Visual Track Embedding from videos and Textual Character Embedding from text context. These two representations are learned to retain rich semantic multimodal information that enables even simple MLPs to achieve the state-of-the-art performance on the target tasks. More specifically, our CiSIN model achieves the best performance in the Fill-in the Characters task of LSMDC 2019 challenges[35]. Moreover, it outperforms previous state-of-the-art models in M-VAD Names dataset [30] as a benchmark of multimodal character grounding and re-identification.

UR - http://www.scopus.com/inward/record.url?scp=85097365586&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85097365586&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-58558-7_32

DO - 10.1007/978-3-030-58558-7_32

M3 - Conference contribution

AN - SCOPUS:85097365586

SN - 9783030585570

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 543

EP - 559

BT - Computer Vision – ECCV 2020 - 16th European Conference, Proceedings

A2 - Vedaldi, Andrea

A2 - Bischof, Horst

A2 - Brox, Thomas

A2 - Frahm, Jan-Michael

PB - Springer Science and Business Media Deutschland GmbH

T2 - 16th European Conference on Computer Vision, ECCV 2020

Y2 - 23 August 2020 through 28 August 2020

ER -

Yu Y, Kim J, Yun H, Chung J, Kim G. Character Grounding and Re-identification in Story of Videos and Text Descriptions. In Vedaldi A, Bischof H, Brox T, Frahm J-M, editors, Computer Vision – ECCV 2020 - 16th European Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 543-559. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-58558-7_32

Character Grounding and Re-identification in Story of Videos and Text Descriptions

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this