Dynamic scene blur is usually caused by object motion, depth variation as well as camera shake. Most existing methods usually solve this problem using image segmentation or fully end-to-end trainable deep convolutional neural networks by considering different object motions or camera shakes. However, these algorithms are less effective when there exist depth variations. In this work, we propose a deep neural convolutional network that exploits the depth map for dynamic scene deblurring. Given a blurred image, we first extract the depth map and adopt a depth refinement network to restore the edges and structure in the depth map. To effectively exploit the depth map, we adopt the spatial feature transform layer to extract depth features and fuse with the image features through scaling and shifting. Our image deblurring network thus learns to restore a clear image under the guidance of the depth map. With substantial experiments and analysis, we show that the depth information is crucial to the performance of the proposed model. Finally, extensive quantitative and qualitative evaluations demonstrate that the proposed model performs favorably against the state-of-the-art dynamic scene deblurring approaches as well as conventional depth-based deblurring algorithms.
Bibliographical noteFunding Information:
Manuscript received March 28, 2019; revised October 12, 2019, January 5, 2020, and February 26, 2020; accepted February 26, 2020. Date of publication March 20, 2020; date of current version March 27, 2020. This work was supported in part by the National Natural Science Foundation of China under Grant 61433007, Grant 61872421, and Grant 61922043, in part by the Natural Science Foundation of Jiangsu Province under Grant BK20180471, and in part by the National Science Foundation CAREER under Grant 1149783. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Zhen He. (Corresponding author: Nong Sang.) Lerenhan Li, Changxin Gao, and Nong Sang are with the National Key Laboratory of Science and Technology on Multispectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China (e-mail: firstname.lastname@example.org; email@example.com; firstname.lastname@example.org).
© 2020 IEEE.
All Science Journal Classification (ASJC) codes
- Computer Graphics and Computer-Aided Design