Factorizing Perception and Policy for Interactive Instruction Following

Kunal Pratap Singh; Suvaansh Bhambri; Byeonghwi Kim; Roozbeh Mottaghi; Jonghyun Choi

doi:10.1109/ICCV48922.2021.00190

Factorizing Perception and Policy for Interactive Instruction Following

Kunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, Jonghyun Choi

Department of Artificial Intelligence

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Citations (Scopus)

Abstract

Performing simple household tasks based on language directives is very natural to humans, yet it remains an open challenge for AI agents. The 'interactive instruction following' task attempts to make progress towards building agents that jointly navigate, interact, and reason in the environment at every step. To address the multifaceted problem, we propose a model that factorizes the task into interactive perception and action policy streams with enhanced components and name it as MOCA, a Modular Object-Centric Approach. We empirically validate that MOCA outperforms prior arts by significant margins on the ALFRED benchmark with improved generalization.

Original language	English
Title of host publication	Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1868-1877
Number of pages	10
ISBN (Electronic)	9781665428125
DOIs	https://doi.org/10.1109/ICCV48922.2021.00190
Publication status	Published - 2021
Event	18th IEEE/CVF International Conference on Computer Vision, ICCV 2021 - Virtual, Online, Canada Duration: 2021 Oct 11 → 2021 Oct 17

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
ISSN (Print)	1550-5499

Conference

Conference	18th IEEE/CVF International Conference on Computer Vision, ICCV 2021
Country/Territory	Canada
City	Virtual, Online
Period	21/10/11 → 21/10/17

Bibliographical note

Funding Information:
Acknowledgement. This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2019R1C1C1009283) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01842, Artificial Intelligence Graduate School Program (GIST)), (No.2019-0-01351, Development of Ultra Low-Power Mobile Deep Learning Semiconductor With Compression/Decompression of Activation/Kernel Data, 25%) and (No. 2021-0-02068, Artificial Intelligence Innovation Hub).

Publisher Copyright:
© 2021 IEEE

All Science Journal Classification (ASJC) codes

Software
Computer Vision and Pattern Recognition

Access to Document

10.1109/ICCV48922.2021.00190

Cite this

Singh, K. P., Bhambri, S., Kim, B., Mottaghi, R., & Choi, J. (2021). Factorizing Perception and Policy for Interactive Instruction Following. In Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021 (pp. 1868-1877). (Proceedings of the IEEE International Conference on Computer Vision). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCV48922.2021.00190

@inproceedings{5fd05b49b34e4f4db914714bac0e91a8,

title = "Factorizing Perception and Policy for Interactive Instruction Following",

abstract = "Performing simple household tasks based on language directives is very natural to humans, yet it remains an open challenge for AI agents. The 'interactive instruction following' task attempts to make progress towards building agents that jointly navigate, interact, and reason in the environment at every step. To address the multifaceted problem, we propose a model that factorizes the task into interactive perception and action policy streams with enhanced components and name it as MOCA, a Modular Object-Centric Approach. We empirically validate that MOCA outperforms prior arts by significant margins on the ALFRED benchmark with improved generalization.",

author = "Singh, {Kunal Pratap} and Suvaansh Bhambri and Byeonghwi Kim and Roozbeh Mottaghi and Jonghyun Choi",

note = "Funding Information: Acknowledgement. This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2019R1C1C1009283) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01842, Artificial Intelligence Graduate School Program (GIST)), (No.2019-0-01351, Development of Ultra Low-Power Mobile Deep Learning Semiconductor With Compression/Decompression of Activation/Kernel Data, 25%) and (No. 2021-0-02068, Artificial Intelligence Innovation Hub). Publisher Copyright: {\textcopyright} 2021 IEEE; 18th IEEE/CVF International Conference on Computer Vision, ICCV 2021 ; Conference date: 11-10-2021 Through 17-10-2021",

year = "2021",

doi = "10.1109/ICCV48922.2021.00190",

language = "English",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1868--1877",

booktitle = "Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021",

address = "United States",

}

Singh, KP, Bhambri, S, Kim, B, Mottaghi, R & Choi, J 2021, Factorizing Perception and Policy for Interactive Instruction Following. in Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021. Proceedings of the IEEE International Conference on Computer Vision, Institute of Electrical and Electronics Engineers Inc., pp. 1868-1877, 18th IEEE/CVF International Conference on Computer Vision, ICCV 2021, Virtual, Online, Canada, 21/10/11. https://doi.org/10.1109/ICCV48922.2021.00190

Factorizing Perception and Policy for Interactive Instruction Following. / Singh, Kunal Pratap; Bhambri, Suvaansh; Kim, Byeonghwi et al.

Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021. Institute of Electrical and Electronics Engineers Inc., 2021. p. 1868-1877 (Proceedings of the IEEE International Conference on Computer Vision).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Factorizing Perception and Policy for Interactive Instruction Following

AU - Singh, Kunal Pratap

AU - Bhambri, Suvaansh

AU - Kim, Byeonghwi

AU - Mottaghi, Roozbeh

AU - Choi, Jonghyun

N1 - Funding Information: Acknowledgement. This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2019R1C1C1009283) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01842, Artificial Intelligence Graduate School Program (GIST)), (No.2019-0-01351, Development of Ultra Low-Power Mobile Deep Learning Semiconductor With Compression/Decompression of Activation/Kernel Data, 25%) and (No. 2021-0-02068, Artificial Intelligence Innovation Hub). Publisher Copyright: © 2021 IEEE

PY - 2021

Y1 - 2021

N2 - Performing simple household tasks based on language directives is very natural to humans, yet it remains an open challenge for AI agents. The 'interactive instruction following' task attempts to make progress towards building agents that jointly navigate, interact, and reason in the environment at every step. To address the multifaceted problem, we propose a model that factorizes the task into interactive perception and action policy streams with enhanced components and name it as MOCA, a Modular Object-Centric Approach. We empirically validate that MOCA outperforms prior arts by significant margins on the ALFRED benchmark with improved generalization.

AB - Performing simple household tasks based on language directives is very natural to humans, yet it remains an open challenge for AI agents. The 'interactive instruction following' task attempts to make progress towards building agents that jointly navigate, interact, and reason in the environment at every step. To address the multifaceted problem, we propose a model that factorizes the task into interactive perception and action policy streams with enhanced components and name it as MOCA, a Modular Object-Centric Approach. We empirically validate that MOCA outperforms prior arts by significant margins on the ALFRED benchmark with improved generalization.

UR - http://www.scopus.com/inward/record.url?scp=85127755645&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85127755645&partnerID=8YFLogxK

U2 - 10.1109/ICCV48922.2021.00190

DO - 10.1109/ICCV48922.2021.00190

M3 - Conference contribution

AN - SCOPUS:85127755645

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 1868

EP - 1877

BT - Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 18th IEEE/CVF International Conference on Computer Vision, ICCV 2021

Y2 - 11 October 2021 through 17 October 2021

ER -

Singh KP, Bhambri S, Kim B, Mottaghi R, Choi J. Factorizing Perception and Policy for Interactive Instruction Following. In Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 1868-1877. (Proceedings of the IEEE International Conference on Computer Vision). doi: 10.1109/ICCV48922.2021.00190

Factorizing Perception and Policy for Interactive Instruction Following

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this