Three dimensional colorization based image/video reconstruction from white-dominant RGBW pattern images

TY - JOUR

T1 - Three dimensional colorization based image/video reconstruction from white-dominant RGBW pattern images

AU - Lee, Suk ho

AU - Oh, Paul

AU - Kang, Moon Gi

PY - 2019/10/1

Y1 - 2019/10/1

N2 - RGBW (RGB-White) CFAs (Color Filter Arrays) which contain white pixels as well as RGB pixels have been actively proposed, which is due to the higher SNR (Signal to Noise Ratio) characteristics of the white pixels. The higher SNR characteristics of the white pixels results in the reconstruction of color images with higher resolutions than those reconstructed with widely used RGB CFAs, especially in low illumination environments. Recently, we proposed a new RGBW demosaicking method which uses an RGBW CFA where the white pixels cover 75% of the CFA, and which reconstructs the color image from the RGBW pattern image by use of the colorization method. The use of a large number of white pixels, and the use of colorization as an interpolation method made it possible to reconstruct a color image with high SNR values. In this paper, we extend this method to reconstruct video sequences by extending the colorization method to the 3-D domain, i.e., we propose a 3-D color reconstruction method which takes an RGBW pattern video sequence as the input and produces a reconstructed color video sequence. We implement the colorization as a 3-D diffusion instead of solving an overall colorization matrix, since the computation is high for the 3-D domain. The 3-D diffusion flow stops at temporal edges, which prevents the false colorizing of moving objects. Furthermore, the 3-D diffusion flow also eliminates the incoherence between colors of neighboring frames, which eliminates the flickering artifact. One major difficulty in extending to the temporal axis is that the noise in the few color seeds can propagate throughout the 3-D domain resulting in many false colors. This is especially important in the case when the image/video is taken in low illumination, since the energy of the thermal noise becomes relatively large compared to the light energy making the RGBW pattern image/video very noisy. To prevent the propagation of false colors caused by the noise in the color seeds, we use the 3-D stacked white channels as a guidance for denoising, so that the denoising takes into account only the color seeds that belong to the same object region in the 3-D space. The stacked white channels also guide the 3-D diffusion throughout the 3-D space. Experimental results show that the image/video reconstructed by the proposed method has better visual quality than those reconstructed by conventional RGB/RGBW CFA based methods. Therefore, the proposed method can be used in surveillance camera applications to recover important information from noisy video sequences taken in very low illumination environment.

AB - RGBW (RGB-White) CFAs (Color Filter Arrays) which contain white pixels as well as RGB pixels have been actively proposed, which is due to the higher SNR (Signal to Noise Ratio) characteristics of the white pixels. The higher SNR characteristics of the white pixels results in the reconstruction of color images with higher resolutions than those reconstructed with widely used RGB CFAs, especially in low illumination environments. Recently, we proposed a new RGBW demosaicking method which uses an RGBW CFA where the white pixels cover 75% of the CFA, and which reconstructs the color image from the RGBW pattern image by use of the colorization method. The use of a large number of white pixels, and the use of colorization as an interpolation method made it possible to reconstruct a color image with high SNR values. In this paper, we extend this method to reconstruct video sequences by extending the colorization method to the 3-D domain, i.e., we propose a 3-D color reconstruction method which takes an RGBW pattern video sequence as the input and produces a reconstructed color video sequence. We implement the colorization as a 3-D diffusion instead of solving an overall colorization matrix, since the computation is high for the 3-D domain. The 3-D diffusion flow stops at temporal edges, which prevents the false colorizing of moving objects. Furthermore, the 3-D diffusion flow also eliminates the incoherence between colors of neighboring frames, which eliminates the flickering artifact. One major difficulty in extending to the temporal axis is that the noise in the few color seeds can propagate throughout the 3-D domain resulting in many false colors. This is especially important in the case when the image/video is taken in low illumination, since the energy of the thermal noise becomes relatively large compared to the light energy making the RGBW pattern image/video very noisy. To prevent the propagation of false colors caused by the noise in the color seeds, we use the 3-D stacked white channels as a guidance for denoising, so that the denoising takes into account only the color seeds that belong to the same object region in the 3-D space. The stacked white channels also guide the 3-D diffusion throughout the 3-D space. Experimental results show that the image/video reconstructed by the proposed method has better visual quality than those reconstructed by conventional RGB/RGBW CFA based methods. Therefore, the proposed method can be used in surveillance camera applications to recover important information from noisy video sequences taken in very low illumination environment.

UR - http://www.scopus.com/inward/record.url?scp=85069829836&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85069829836&partnerID=8YFLogxK

U2 - 10.1016/j.dsp.2019.07.012

DO - 10.1016/j.dsp.2019.07.012

M3 - Article

VL - 93

SP - 87

EP - 101

JO - Digital Signal Processing: A Review Journal

JF - Digital Signal Processing: A Review Journal

SN - 1051-2004

ER -