Fast intermode decision algorithm based on general and local residual complexity in H.264/AVC

The state-of-the-art video coding standard H.264/AVC achieves significant coding performance by adopting variable block sizes for motion estimation (ME) and mode decision. However, this technique brings out high computational complexity since the optimal mode is determined by exhaustively performing rate-distortion optimization (RDO) on each coding mode with different block sizes. In this paper, the fast intermode decision algorithm is proposed to reduce the computational complexity. Based on the high correlation between the residual error of ME and the optimal block size, general residual complexity (GRC) and local residual complexity (LRC) are defined. According to MB activity evaluated on GRC and LRC, candidate intermodes are determined and RDO processes are only performed on selected intermodes. The experimental results demonstrate that the proposed algorithm achieves time saving by 63% on average with negligible degradation of coding efficiency.