Query generation for multimodal documents

Kyungho Kim, Kyungjae Lee, Seung Won Hwang, Young In Song, Seungwook Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper studies the problem of generating likely queries for multimodal documents with images. Our application scenario is enabling efficient “first-stage retrieval” of relevant documents, by attaching generated queries to documents before indexing. We can then index this expanded text to efficiently narrow down to candidate matches using inverted index, so that expensive reranking can follow. Our evaluation results show that our proposed multimodal representation meaningfully improves relevance ranking. More importantly, our framework can achieve the state of the art in the first-stage retrieval scenarios.

Original languageEnglish
Title of host publicationEACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference
PublisherAssociation for Computational Linguistics (ACL)
Pages659-668
Number of pages10
ISBN (Electronic)9781954085022
Publication statusPublished - 2021
Event16th Conference of the European Chapter of the Associationfor Computational Linguistics, EACL 2021 - Virtual, Online
Duration: 2021 Apr 192021 Apr 23

Publication series

NameEACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

Conference

Conference16th Conference of the European Chapter of the Associationfor Computational Linguistics, EACL 2021
CityVirtual, Online
Period21/4/1921/4/23

Bibliographical note

Funding Information:
This research was supported by the MSIT, under IITP-2017-0-01779; A machine learning and statistical inference framework for explainable artificial intelligence) and the ITRC support program (IITP-2021-2020-0-01789), supervised by the IITP.

Publisher Copyright:
© 2021 Association for Computational Linguistics

All Science Journal Classification (ASJC) codes

  • Software
  • Computational Theory and Mathematics
  • Linguistics and Language

Fingerprint

Dive into the research topics of 'Query generation for multimodal documents'. Together they form a unique fingerprint.

Cite this