Video-text compliance: Activity verification based on natural language instructions

Mayoore Jaiswal; Frank Liu; Anupama Jagannathan; Anne Gattiker; Inseok Hwang; Jinho Lee; Matthew Tong; Sahil Dureja; Soham Shah; Peter Hofstee; Valerie Chen; Suvadip Paul; Rogerio Feris

doi:10.1109/ICCVW.2019.00188

Video-text compliance: Activity verification based on natural language instructions

Mayoore Jaiswal, Frank Liu, Anupama Jagannathan, Anne Gattiker, Inseok Hwang, Jinho Lee, Matthew Tong, Sahil Dureja, Soham Shah, Peter Hofstee, Valerie Chen, Suvadip Paul, Rogerio Feris

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We define a new multi-modal compliance problem, which is to determine if the human activity in a given video is in compliance with an associated text instruction. Solutions to the compliance problem could enable automatic compliance checking and efficient feedback in many real-world settings. To this end, we introduce the Video-Text Compliance (VTC) dataset, which contains videos of atomic activities, along with text instructions and compliance labels. The VTC dataset is constructed by an auto-augmentation technique, preserves privacy, and contains over 1.2 million frames. Finally, we present ComplianceNet, a novel end-to-end trainable compliance network that improves the baseline accuracy by 27.5% on average when trained on the VTC dataset. We plan to release the VTC dataset to the community for future research.

Original language	English
Title of host publication	Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1503-1512
Number of pages	10
ISBN (Electronic)	9781728150239
DOIs	https://doi.org/10.1109/ICCVW.2019.00188
Publication status	Published - 2019 Oct
Event	17th IEEE/CVF International Conference on Computer Vision Workshop, ICCVW 2019 - Seoul, Korea, Republic of Duration: 2019 Oct 27 → 2019 Oct 28

Publication series

Name	Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019

Conference

Conference	17th IEEE/CVF International Conference on Computer Vision Workshop, ICCVW 2019
Country/Territory	Korea, Republic of
City	Seoul
Period	19/10/27 → 19/10/28

Bibliographical note

Publisher Copyright:
© 2019 IEEE.

All Science Journal Classification (ASJC) codes

Computer Science Applications
Computer Vision and Pattern Recognition

Access to Document

10.1109/ICCVW.2019.00188

Cite this

Jaiswal, M., Liu, F., Jagannathan, A., Gattiker, A., Hwang, I., Lee, J., Tong, M., Dureja, S., Shah, S., Hofstee, P., Chen, V., Paul, S., & Feris, R. (2019). Video-text compliance: Activity verification based on natural language instructions. In Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019 (pp. 1503-1512). [9022124] (Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCVW.2019.00188

Jaiswal, Mayoore ; Liu, Frank ; Jagannathan, Anupama et al. / Video-text compliance : Activity verification based on natural language instructions. Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 1503-1512 (Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019).

@inproceedings{bdf6dbfde33e4e828f923321325f29be,

title = "Video-text compliance: Activity verification based on natural language instructions",

abstract = "We define a new multi-modal compliance problem, which is to determine if the human activity in a given video is in compliance with an associated text instruction. Solutions to the compliance problem could enable automatic compliance checking and efficient feedback in many real-world settings. To this end, we introduce the Video-Text Compliance (VTC) dataset, which contains videos of atomic activities, along with text instructions and compliance labels. The VTC dataset is constructed by an auto-augmentation technique, preserves privacy, and contains over 1.2 million frames. Finally, we present ComplianceNet, a novel end-to-end trainable compliance network that improves the baseline accuracy by 27.5% on average when trained on the VTC dataset. We plan to release the VTC dataset to the community for future research.",

author = "Mayoore Jaiswal and Frank Liu and Anupama Jagannathan and Anne Gattiker and Inseok Hwang and Jinho Lee and Matthew Tong and Sahil Dureja and Soham Shah and Peter Hofstee and Valerie Chen and Suvadip Paul and Rogerio Feris",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 17th IEEE/CVF International Conference on Computer Vision Workshop, ICCVW 2019 ; Conference date: 27-10-2019 Through 28-10-2019",

year = "2019",

month = oct,

doi = "10.1109/ICCVW.2019.00188",

language = "English",

series = "Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1503--1512",

booktitle = "Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019",

address = "United States",

}

Jaiswal, M, Liu, F, Jagannathan, A, Gattiker, A, Hwang, I, Lee, J, Tong, M, Dureja, S, Shah, S, Hofstee, P, Chen, V, Paul, S & Feris, R 2019, Video-text compliance: Activity verification based on natural language instructions. in Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019., 9022124, Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019, Institute of Electrical and Electronics Engineers Inc., pp. 1503-1512, 17th IEEE/CVF International Conference on Computer Vision Workshop, ICCVW 2019, Seoul, Korea, Republic of, 19/10/27. https://doi.org/10.1109/ICCVW.2019.00188

Video-text compliance : Activity verification based on natural language instructions. / Jaiswal, Mayoore; Liu, Frank; Jagannathan, Anupama et al.

Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 1503-1512 9022124 (Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Video-text compliance

T2 - 17th IEEE/CVF International Conference on Computer Vision Workshop, ICCVW 2019

AU - Jaiswal, Mayoore

AU - Liu, Frank

AU - Jagannathan, Anupama

AU - Gattiker, Anne

AU - Hwang, Inseok

AU - Lee, Jinho

AU - Tong, Matthew

AU - Dureja, Sahil

AU - Shah, Soham

AU - Hofstee, Peter

AU - Chen, Valerie

AU - Paul, Suvadip

AU - Feris, Rogerio

PY - 2019/10

Y1 - 2019/10

N2 - We define a new multi-modal compliance problem, which is to determine if the human activity in a given video is in compliance with an associated text instruction. Solutions to the compliance problem could enable automatic compliance checking and efficient feedback in many real-world settings. To this end, we introduce the Video-Text Compliance (VTC) dataset, which contains videos of atomic activities, along with text instructions and compliance labels. The VTC dataset is constructed by an auto-augmentation technique, preserves privacy, and contains over 1.2 million frames. Finally, we present ComplianceNet, a novel end-to-end trainable compliance network that improves the baseline accuracy by 27.5% on average when trained on the VTC dataset. We plan to release the VTC dataset to the community for future research.

AB - We define a new multi-modal compliance problem, which is to determine if the human activity in a given video is in compliance with an associated text instruction. Solutions to the compliance problem could enable automatic compliance checking and efficient feedback in many real-world settings. To this end, we introduce the Video-Text Compliance (VTC) dataset, which contains videos of atomic activities, along with text instructions and compliance labels. The VTC dataset is constructed by an auto-augmentation technique, preserves privacy, and contains over 1.2 million frames. Finally, we present ComplianceNet, a novel end-to-end trainable compliance network that improves the baseline accuracy by 27.5% on average when trained on the VTC dataset. We plan to release the VTC dataset to the community for future research.

UR - http://www.scopus.com/inward/record.url?scp=85082503092&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85082503092&partnerID=8YFLogxK

U2 - 10.1109/ICCVW.2019.00188

DO - 10.1109/ICCVW.2019.00188

M3 - Conference contribution

AN - SCOPUS:85082503092

T3 - Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019

SP - 1503

EP - 1512

BT - Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 27 October 2019 through 28 October 2019

ER -

Jaiswal M, Liu F, Jagannathan A, Gattiker A, Hwang I, Lee J et al. Video-text compliance: Activity verification based on natural language instructions. In Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 1503-1512. 9022124. (Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019). doi: 10.1109/ICCVW.2019.00188

Video-text compliance: Activity verification based on natural language instructions

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this