SegFlow: Joint Learning for Video Object Segmentation and Optical Flow

Jingchun Cheng, Yi Hsuan Tsai, Shengjin Wang, Ming Hsuan Yang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

87 Citations (Scopus)

Abstract

This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.

Original languageEnglish
Title of host publicationProceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages686-695
Number of pages10
ISBN (Electronic)9781538610329
DOIs
Publication statusPublished - 2017 Dec 22
Event16th IEEE International Conference on Computer Vision, ICCV 2017 - Venice, Italy
Duration: 2017 Oct 222017 Oct 29

Publication series

NameProceedings of the IEEE International Conference on Computer Vision
Volume2017-October
ISSN (Print)1550-5499

Other

Other16th IEEE International Conference on Computer Vision, ICCV 2017
CountryItaly
CityVenice
Period17/10/2217/10/29

Fingerprint

Optical flows
Image segmentation
Pixels
Experiments

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition

Cite this

Cheng, J., Tsai, Y. H., Wang, S., & Yang, M. H. (2017). SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. In Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017 (pp. 686-695). [8237343] (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2017-October). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCV.2017.81
Cheng, Jingchun ; Tsai, Yi Hsuan ; Wang, Shengjin ; Yang, Ming Hsuan. / SegFlow : Joint Learning for Video Object Segmentation and Optical Flow. Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 686-695 (Proceedings of the IEEE International Conference on Computer Vision).
@inproceedings{b59866595c4c40d29e47c9cbb3d2155d,
title = "SegFlow: Joint Learning for Video Object Segmentation and Optical Flow",
abstract = "This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.",
author = "Jingchun Cheng and Tsai, {Yi Hsuan} and Shengjin Wang and Yang, {Ming Hsuan}",
year = "2017",
month = "12",
day = "22",
doi = "10.1109/ICCV.2017.81",
language = "English",
series = "Proceedings of the IEEE International Conference on Computer Vision",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "686--695",
booktitle = "Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017",
address = "United States",

}

Cheng, J, Tsai, YH, Wang, S & Yang, MH 2017, SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. in Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017., 8237343, Proceedings of the IEEE International Conference on Computer Vision, vol. 2017-October, Institute of Electrical and Electronics Engineers Inc., pp. 686-695, 16th IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 17/10/22. https://doi.org/10.1109/ICCV.2017.81

SegFlow : Joint Learning for Video Object Segmentation and Optical Flow. / Cheng, Jingchun; Tsai, Yi Hsuan; Wang, Shengjin; Yang, Ming Hsuan.

Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 686-695 8237343 (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2017-October).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - SegFlow

T2 - Joint Learning for Video Object Segmentation and Optical Flow

AU - Cheng, Jingchun

AU - Tsai, Yi Hsuan

AU - Wang, Shengjin

AU - Yang, Ming Hsuan

PY - 2017/12/22

Y1 - 2017/12/22

N2 - This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.

AB - This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.

UR - http://www.scopus.com/inward/record.url?scp=85041903685&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041903685&partnerID=8YFLogxK

U2 - 10.1109/ICCV.2017.81

DO - 10.1109/ICCV.2017.81

M3 - Conference contribution

AN - SCOPUS:85041903685

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 686

EP - 695

BT - Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Cheng J, Tsai YH, Wang S, Yang MH. SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. In Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 686-695. 8237343. (Proceedings of the IEEE International Conference on Computer Vision). https://doi.org/10.1109/ICCV.2017.81