Video Deblurring via Semantic Segmentation and Pixel-Wise Non-linear Kernel

Wenqi Ren, Jinshan Pan, Xiaochun Cao, Ming Hsuan Yang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

14 Citations (Scopus)

Abstract

Video deblurring is a challenging problem as the blur is complex and usually caused by the combination of camera shakes, object motions, and depth variations. Optical flow can be used for kernel estimation since it predicts motion trajectories. However, the estimates are often inaccurate in complex scenes at object boundaries, which are crucial in kernel estimation. In this paper, we exploit semantic segmentation in each blurry frame to understand the scene contents and use different motion models for image regions to guide optical flow estimation. While existing pixel-wise blur models assume that the blur kernel is the same as optical flow during the exposure time, this assumption does not hold when the motion blur trajectory at a pixel is different from the estimated linear optical flow. We analyze the relationship between motion blur trajectory and optical flow, and present a novel pixel-wise non-linear kernel model to account for motion blur. The proposed blur model is based on the non-linear optical flow, which describes complex motion blur more effectively. Extensive experiments on challenging blurry videos demonstrate the proposed algorithm performs favorably against the state-of-the-art methods.

Original languageEnglish
Title of host publicationProceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1086-1094
Number of pages9
ISBN (Electronic)9781538610329
DOIs
Publication statusPublished - 2017 Dec 22
Event16th IEEE International Conference on Computer Vision, ICCV 2017 - Venice, Italy
Duration: 2017 Oct 222017 Oct 29

Publication series

NameProceedings of the IEEE International Conference on Computer Vision
Volume2017-October
ISSN (Print)1550-5499

Other

Other16th IEEE International Conference on Computer Vision, ICCV 2017
CountryItaly
CityVenice
Period17/10/2217/10/29

Fingerprint

Optical flows
Pixels
Semantics
Trajectories
Cameras
Experiments

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition

Cite this

Ren, W., Pan, J., Cao, X., & Yang, M. H. (2017). Video Deblurring via Semantic Segmentation and Pixel-Wise Non-linear Kernel. In Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017 (pp. 1086-1094). [8237385] (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2017-October). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCV.2017.123
Ren, Wenqi ; Pan, Jinshan ; Cao, Xiaochun ; Yang, Ming Hsuan. / Video Deblurring via Semantic Segmentation and Pixel-Wise Non-linear Kernel. Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 1086-1094 (Proceedings of the IEEE International Conference on Computer Vision).
@inproceedings{610a5c6b97aa4a5d83830630545d9d18,
title = "Video Deblurring via Semantic Segmentation and Pixel-Wise Non-linear Kernel",
abstract = "Video deblurring is a challenging problem as the blur is complex and usually caused by the combination of camera shakes, object motions, and depth variations. Optical flow can be used for kernel estimation since it predicts motion trajectories. However, the estimates are often inaccurate in complex scenes at object boundaries, which are crucial in kernel estimation. In this paper, we exploit semantic segmentation in each blurry frame to understand the scene contents and use different motion models for image regions to guide optical flow estimation. While existing pixel-wise blur models assume that the blur kernel is the same as optical flow during the exposure time, this assumption does not hold when the motion blur trajectory at a pixel is different from the estimated linear optical flow. We analyze the relationship between motion blur trajectory and optical flow, and present a novel pixel-wise non-linear kernel model to account for motion blur. The proposed blur model is based on the non-linear optical flow, which describes complex motion blur more effectively. Extensive experiments on challenging blurry videos demonstrate the proposed algorithm performs favorably against the state-of-the-art methods.",
author = "Wenqi Ren and Jinshan Pan and Xiaochun Cao and Yang, {Ming Hsuan}",
year = "2017",
month = "12",
day = "22",
doi = "10.1109/ICCV.2017.123",
language = "English",
series = "Proceedings of the IEEE International Conference on Computer Vision",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "1086--1094",
booktitle = "Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017",
address = "United States",

}

Ren, W, Pan, J, Cao, X & Yang, MH 2017, Video Deblurring via Semantic Segmentation and Pixel-Wise Non-linear Kernel. in Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017., 8237385, Proceedings of the IEEE International Conference on Computer Vision, vol. 2017-October, Institute of Electrical and Electronics Engineers Inc., pp. 1086-1094, 16th IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 17/10/22. https://doi.org/10.1109/ICCV.2017.123

Video Deblurring via Semantic Segmentation and Pixel-Wise Non-linear Kernel. / Ren, Wenqi; Pan, Jinshan; Cao, Xiaochun; Yang, Ming Hsuan.

Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 1086-1094 8237385 (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2017-October).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Video Deblurring via Semantic Segmentation and Pixel-Wise Non-linear Kernel

AU - Ren, Wenqi

AU - Pan, Jinshan

AU - Cao, Xiaochun

AU - Yang, Ming Hsuan

PY - 2017/12/22

Y1 - 2017/12/22

N2 - Video deblurring is a challenging problem as the blur is complex and usually caused by the combination of camera shakes, object motions, and depth variations. Optical flow can be used for kernel estimation since it predicts motion trajectories. However, the estimates are often inaccurate in complex scenes at object boundaries, which are crucial in kernel estimation. In this paper, we exploit semantic segmentation in each blurry frame to understand the scene contents and use different motion models for image regions to guide optical flow estimation. While existing pixel-wise blur models assume that the blur kernel is the same as optical flow during the exposure time, this assumption does not hold when the motion blur trajectory at a pixel is different from the estimated linear optical flow. We analyze the relationship between motion blur trajectory and optical flow, and present a novel pixel-wise non-linear kernel model to account for motion blur. The proposed blur model is based on the non-linear optical flow, which describes complex motion blur more effectively. Extensive experiments on challenging blurry videos demonstrate the proposed algorithm performs favorably against the state-of-the-art methods.

AB - Video deblurring is a challenging problem as the blur is complex and usually caused by the combination of camera shakes, object motions, and depth variations. Optical flow can be used for kernel estimation since it predicts motion trajectories. However, the estimates are often inaccurate in complex scenes at object boundaries, which are crucial in kernel estimation. In this paper, we exploit semantic segmentation in each blurry frame to understand the scene contents and use different motion models for image regions to guide optical flow estimation. While existing pixel-wise blur models assume that the blur kernel is the same as optical flow during the exposure time, this assumption does not hold when the motion blur trajectory at a pixel is different from the estimated linear optical flow. We analyze the relationship between motion blur trajectory and optical flow, and present a novel pixel-wise non-linear kernel model to account for motion blur. The proposed blur model is based on the non-linear optical flow, which describes complex motion blur more effectively. Extensive experiments on challenging blurry videos demonstrate the proposed algorithm performs favorably against the state-of-the-art methods.

UR - http://www.scopus.com/inward/record.url?scp=85041903240&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041903240&partnerID=8YFLogxK

U2 - 10.1109/ICCV.2017.123

DO - 10.1109/ICCV.2017.123

M3 - Conference contribution

AN - SCOPUS:85041903240

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 1086

EP - 1094

BT - Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Ren W, Pan J, Cao X, Yang MH. Video Deblurring via Semantic Segmentation and Pixel-Wise Non-linear Kernel. In Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 1086-1094. 8237385. (Proceedings of the IEEE International Conference on Computer Vision). https://doi.org/10.1109/ICCV.2017.123