Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension

Aniruddha Kembhavi; Minjoon Seo; Dustin Schwenk; Jonghyun Choi; Ali Farhadi; Hannaneh Hajishirzi

doi:10.1109/CVPR.2017.571

Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension

Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, Hannaneh Hajishirzi

Department of Artificial Intelligence

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

78 Citations (Scopus)

Abstract

We introduce the task of Multi-Modal Machine Comprehension (M³C), which aims at answering multimodal questions given a context of text, diagrams and images. We present the Textbook Question Answering (TQA) dataset that includes 1,076 lessons and 26,260 multi-modal questions, taken from middle school science curricula. Our analysis shows that a significant portion of questions require complex parsing of the text and the diagrams and reasoning, indicating that our dataset is more complex compared to previous machine comprehension and visual question answering datasets. We extend state-of-the-art methods for textual machine comprehension and visual question answering to the TQA dataset. Our experiments show that these models do not perform well on TQA. The presented dataset opens new challenges for research in question answering and reasoning across multiple modalities.

Original language	English
Title of host publication	Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	5376-5384
Number of pages	9
ISBN (Electronic)	9781538604571
DOIs	https://doi.org/10.1109/CVPR.2017.571
Publication status	Published - 2017 Nov 6
Event	30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 - Honolulu, United States Duration: 2017 Jul 21 → 2017 Jul 26

Publication series

Name	Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
Volume	2017-January

Other

Other	30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
Country/Territory	United States
City	Honolulu
Period	17/7/21 → 17/7/26

Bibliographical note

Funding Information:
This work is in part supported by ONR N00014-13-1-0720, NSF IIS-1338054, NSF-1652052, NRI-1637479, NSF IIS-1616112, Allen Distinguished Investigator Award, Google Research Faculty Award, Samsung GRO Award and the Allen Institute for Artificial Intelligence.

Publisher Copyright:
© 2017 IEEE.

All Science Journal Classification (ASJC) codes

Signal Processing
Computer Vision and Pattern Recognition

Access to Document

10.1109/CVPR.2017.571

Cite this

Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017). Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (pp. 5376-5384). (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017; Vol. 2017-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CVPR.2017.571

Kembhavi, Aniruddha ; Seo, Minjoon ; Schwenk, Dustin ; Choi, Jonghyun ; Farhadi, Ali ; Hajishirzi, Hannaneh. / Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 5376-5384 (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017).

@inproceedings{963dad74ff4f42fbb1295eaf2aa4689f,

title = "Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension",

abstract = "We introduce the task of Multi-Modal Machine Comprehension (M3C), which aims at answering multimodal questions given a context of text, diagrams and images. We present the Textbook Question Answering (TQA) dataset that includes 1,076 lessons and 26,260 multi-modal questions, taken from middle school science curricula. Our analysis shows that a significant portion of questions require complex parsing of the text and the diagrams and reasoning, indicating that our dataset is more complex compared to previous machine comprehension and visual question answering datasets. We extend state-of-the-art methods for textual machine comprehension and visual question answering to the TQA dataset. Our experiments show that these models do not perform well on TQA. The presented dataset opens new challenges for research in question answering and reasoning across multiple modalities.",

author = "Aniruddha Kembhavi and Minjoon Seo and Dustin Schwenk and Jonghyun Choi and Ali Farhadi and Hannaneh Hajishirzi",

note = "Funding Information: This work is in part supported by ONR N00014-13-1-0720, NSF IIS-1338054, NSF-1652052, NRI-1637479, NSF IIS-1616112, Allen Distinguished Investigator Award, Google Research Faculty Award, Samsung GRO Award and the Allen Institute for Artificial Intelligence. Publisher Copyright: {\textcopyright} 2017 IEEE.; 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 ; Conference date: 21-07-2017 Through 26-07-2017",

year = "2017",

month = nov,

day = "6",

doi = "10.1109/CVPR.2017.571",

language = "English",

series = "Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "5376--5384",

booktitle = "Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017",

address = "United States",

}

Kembhavi, A, Seo, M, Schwenk, D, Choi, J, Farhadi, A & Hajishirzi, H 2017, Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension. in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, vol. 2017-January, Institute of Electrical and Electronics Engineers Inc., pp. 5376-5384, 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, United States, 17/7/21. https://doi.org/10.1109/CVPR.2017.571

Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension. / Kembhavi, Aniruddha; Seo, Minjoon; Schwenk, Dustin; Choi, Jonghyun; Farhadi, Ali; Hajishirzi, Hannaneh.

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 5376-5384 (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017; Vol. 2017-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension

AU - Kembhavi, Aniruddha

AU - Seo, Minjoon

AU - Schwenk, Dustin

AU - Choi, Jonghyun

AU - Farhadi, Ali

AU - Hajishirzi, Hannaneh

N1 - Funding Information: This work is in part supported by ONR N00014-13-1-0720, NSF IIS-1338054, NSF-1652052, NRI-1637479, NSF IIS-1616112, Allen Distinguished Investigator Award, Google Research Faculty Award, Samsung GRO Award and the Allen Institute for Artificial Intelligence. Publisher Copyright: © 2017 IEEE.

PY - 2017/11/6

Y1 - 2017/11/6

N2 - We introduce the task of Multi-Modal Machine Comprehension (M3C), which aims at answering multimodal questions given a context of text, diagrams and images. We present the Textbook Question Answering (TQA) dataset that includes 1,076 lessons and 26,260 multi-modal questions, taken from middle school science curricula. Our analysis shows that a significant portion of questions require complex parsing of the text and the diagrams and reasoning, indicating that our dataset is more complex compared to previous machine comprehension and visual question answering datasets. We extend state-of-the-art methods for textual machine comprehension and visual question answering to the TQA dataset. Our experiments show that these models do not perform well on TQA. The presented dataset opens new challenges for research in question answering and reasoning across multiple modalities.

AB - We introduce the task of Multi-Modal Machine Comprehension (M3C), which aims at answering multimodal questions given a context of text, diagrams and images. We present the Textbook Question Answering (TQA) dataset that includes 1,076 lessons and 26,260 multi-modal questions, taken from middle school science curricula. Our analysis shows that a significant portion of questions require complex parsing of the text and the diagrams and reasoning, indicating that our dataset is more complex compared to previous machine comprehension and visual question answering datasets. We extend state-of-the-art methods for textual machine comprehension and visual question answering to the TQA dataset. Our experiments show that these models do not perform well on TQA. The presented dataset opens new challenges for research in question answering and reasoning across multiple modalities.

UR - http://www.scopus.com/inward/record.url?scp=85044457212&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85044457212&partnerID=8YFLogxK

U2 - 10.1109/CVPR.2017.571

DO - 10.1109/CVPR.2017.571

M3 - Conference contribution

AN - SCOPUS:85044457212

T3 - Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

SP - 5376

EP - 5384

BT - Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

Y2 - 21 July 2017 through 26 July 2017

ER -

Kembhavi A, Seo M, Schwenk D, Choi J, Farhadi A, Hajishirzi H. Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 5376-5384. (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017). https://doi.org/10.1109/CVPR.2017.571

Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension

Abstract

Publication series

Other

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this