VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation

Su Ho Han; Sukjun Hwang; Seoung Wug Oh; Yeonchool Park; Hyunwoo Kim; Min Jung Kim; Seon Joo Kim

doi:10.1109/CVPR52688.2022.00291

VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation

Su Ho Han, Sukjun Hwang, Seoung Wug Oh, Yeonchool Park, Hyunwoo Kim, Min Jung Kim, Seon Joo Kim

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

For online video instance segmentation (VIS), fully utilizing the information from previous frames in an efficient manner is essential for real-time applications. Most previous methods follow a two-stage approach requiring additional computations such as RPN and RoIAlign, and do not fully exploit the available information in the video for all subtasks in VIS. In this paper, we propose a novel single-stage framework for online VIS built based on the grid structured feature representation. The grid-based features allow us to employ fully convolutional networks for real-time processing, and also to easily reuse and share features within different components. We also introduce cooperatively operating modules that aggregate information from available frames, in order to enrich the features for all subtasks in VIS. Our design fully takes advantage of previous information in a grid form for all tasks in VIS in an efficient way, and we achieved the new state-of-the-art accuracy (38.6 AP and 36.9 AP) and speed (40.0 FPS) on YouTube-VIS 2019 and 2021 datasets among online VIS methods. The code is available at https://github.com/SuHoHan95/VISOLQ

Original language	English
Title of host publication	Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
Publisher	IEEE Computer Society
Pages	2886-2895
Number of pages	10
ISBN (Electronic)	9781665469463
DOIs	https://doi.org/10.1109/CVPR52688.2022.00291
Publication status	Published - 2022
Event	2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 - New Orleans, United States Duration: 2022 Jun 19 → 2022 Jun 24

Publication series

Name	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume	2022-June
ISSN (Print)	1063-6919

Conference

Conference	2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
Country/Territory	United States
City	New Orleans
Period	22/6/19 → 22/6/24

Bibliographical note

Funding Information:
This research was supported by LG Electronics, and also by Institute of Information & communications Technology Planning & evaluation (IITP) grant funded by the Korea government(MSIT), Artificial Intelligence Graduate School Program, Yonsei University, under Grant 2020-0-01361.

Publisher Copyright:
© 2022 IEEE.

All Science Journal Classification (ASJC) codes

Software
Computer Vision and Pattern Recognition

Access to Document

10.1109/CVPR52688.2022.00291

Cite this

Han, S. H., Hwang, S., Oh, S. W., Park, Y., Kim, H., Kim, M. J., & Kim, S. J. (2022). VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation. In Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 (pp. 2886-2895). (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 2022-June). IEEE Computer Society. https://doi.org/10.1109/CVPR52688.2022.00291

Han, Su Ho ; Hwang, Sukjun ; Oh, Seoung Wug et al. / VISOLO : Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation. Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. IEEE Computer Society, 2022. pp. 2886-2895 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition).

@inproceedings{b1aea1713e81493c8946e3df760b8bc3,

title = "VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation",

abstract = "For online video instance segmentation (VIS), fully utilizing the information from previous frames in an efficient manner is essential for real-time applications. Most previous methods follow a two-stage approach requiring additional computations such as RPN and RoIAlign, and do not fully exploit the available information in the video for all subtasks in VIS. In this paper, we propose a novel single-stage framework for online VIS built based on the grid structured feature representation. The grid-based features allow us to employ fully convolutional networks for real-time processing, and also to easily reuse and share features within different components. We also introduce cooperatively operating modules that aggregate information from available frames, in order to enrich the features for all subtasks in VIS. Our design fully takes advantage of previous information in a grid form for all tasks in VIS in an efficient way, and we achieved the new state-of-the-art accuracy (38.6 AP and 36.9 AP) and speed (40.0 FPS) on YouTube-VIS 2019 and 2021 datasets among online VIS methods. The code is available at https://github.com/SuHoHan95/VISOLQ",

author = "Han, {Su Ho} and Sukjun Hwang and Oh, {Seoung Wug} and Yeonchool Park and Hyunwoo Kim and Kim, {Min Jung} and Kim, {Seon Joo}",

note = "Funding Information: This research was supported by LG Electronics, and also by Institute of Information & communications Technology Planning & evaluation (IITP) grant funded by the Korea government(MSIT), Artificial Intelligence Graduate School Program, Yonsei University, under Grant 2020-0-01361. Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 ; Conference date: 19-06-2022 Through 24-06-2022",

year = "2022",

doi = "10.1109/CVPR52688.2022.00291",

language = "English",

series = "Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition",

publisher = "IEEE Computer Society",

pages = "2886--2895",

booktitle = "Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022",

address = "United States",

}

Han, SH, Hwang, S, Oh, SW, Park, Y, Kim, H, Kim, MJ & Kim, SJ 2022, VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation. in Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2022-June, IEEE Computer Society, pp. 2886-2895, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, United States, 22/6/19. https://doi.org/10.1109/CVPR52688.2022.00291

VISOLO : Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation. / Han, Su Ho; Hwang, Sukjun; Oh, Seoung Wug et al.

Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. IEEE Computer Society, 2022. p. 2886-2895 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 2022-June).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - VISOLO

T2 - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022

AU - Han, Su Ho

AU - Hwang, Sukjun

AU - Oh, Seoung Wug

AU - Park, Yeonchool

AU - Kim, Hyunwoo

AU - Kim, Min Jung

AU - Kim, Seon Joo

N1 - Funding Information: This research was supported by LG Electronics, and also by Institute of Information & communications Technology Planning & evaluation (IITP) grant funded by the Korea government(MSIT), Artificial Intelligence Graduate School Program, Yonsei University, under Grant 2020-0-01361. Publisher Copyright: © 2022 IEEE.

PY - 2022

Y1 - 2022

N2 - For online video instance segmentation (VIS), fully utilizing the information from previous frames in an efficient manner is essential for real-time applications. Most previous methods follow a two-stage approach requiring additional computations such as RPN and RoIAlign, and do not fully exploit the available information in the video for all subtasks in VIS. In this paper, we propose a novel single-stage framework for online VIS built based on the grid structured feature representation. The grid-based features allow us to employ fully convolutional networks for real-time processing, and also to easily reuse and share features within different components. We also introduce cooperatively operating modules that aggregate information from available frames, in order to enrich the features for all subtasks in VIS. Our design fully takes advantage of previous information in a grid form for all tasks in VIS in an efficient way, and we achieved the new state-of-the-art accuracy (38.6 AP and 36.9 AP) and speed (40.0 FPS) on YouTube-VIS 2019 and 2021 datasets among online VIS methods. The code is available at https://github.com/SuHoHan95/VISOLQ

AB - For online video instance segmentation (VIS), fully utilizing the information from previous frames in an efficient manner is essential for real-time applications. Most previous methods follow a two-stage approach requiring additional computations such as RPN and RoIAlign, and do not fully exploit the available information in the video for all subtasks in VIS. In this paper, we propose a novel single-stage framework for online VIS built based on the grid structured feature representation. The grid-based features allow us to employ fully convolutional networks for real-time processing, and also to easily reuse and share features within different components. We also introduce cooperatively operating modules that aggregate information from available frames, in order to enrich the features for all subtasks in VIS. Our design fully takes advantage of previous information in a grid form for all tasks in VIS in an efficient way, and we achieved the new state-of-the-art accuracy (38.6 AP and 36.9 AP) and speed (40.0 FPS) on YouTube-VIS 2019 and 2021 datasets among online VIS methods. The code is available at https://github.com/SuHoHan95/VISOLQ

UR - http://www.scopus.com/inward/record.url?scp=85136998262&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85136998262&partnerID=8YFLogxK

U2 - 10.1109/CVPR52688.2022.00291

DO - 10.1109/CVPR52688.2022.00291

M3 - Conference contribution

AN - SCOPUS:85136998262

T3 - Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

SP - 2886

EP - 2895

BT - Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022

PB - IEEE Computer Society

Y2 - 19 June 2022 through 24 June 2022

ER -

Han SH, Hwang S, Oh SW, Park Y, Kim H, Kim MJ et al. VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation. In Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. IEEE Computer Society. 2022. p. 2886-2895. (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition). doi: 10.1109/CVPR52688.2022.00291

VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this