MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement

Wenbo Bao, Wei Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, Ming Hsuan Yang

Research output: Contribution to journalArticlepeer-review

17 Citations (Scopus)

Abstract

Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net architecture can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets.

Original languageEnglish
Article number8840983
Pages (from-to)933-948
Number of pages16
JournalIEEE transactions on pattern analysis and machine intelligence
Volume43
Issue number3
DOIs
Publication statusPublished - 2021 Mar 1

Bibliographical note

Funding Information:
W.-S. Lai and M.-H. Yang are supported in part by NSF Career Grant (1149783) and gifts from Adobe, Google, and NEC.

Funding Information:
W. Bao, X. Zhang, and Z. Gao are supported in part by the National Natural Science Foundation of China (61771306), the Natural Science Foundation of Shanghai (18ZR1418100), the Chinese National Key S&T Special Program (2013ZX01033001-002-002), and the Shanghai Key Laboratory of Digital Media Processing and Transmissions (STCSM 18DZ2270700).

Publisher Copyright:
© 1979-2012 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement'. Together they form a unique fingerprint.

Cite this