Fusion of appearance and depth information for face recognition

In this paper, an investigation is carried out regarding combination of 3D and 2D information for face recognition. A two-stage method, PCA and a Reduced Multivariate Polynomial Model (RMPM), is developed to fuse the appearance and depth information of face images in feature level where simplicity (number of polynomial coefficients increases linearly with model-order and input-dimension, i.e. no dimension explosion as in the case of full multivariate polynomials) and ease of use (can be easily formulated into recursive learning fashion) are major concerns. To cater for fast on-line registration capability when a new user arrives, the learning is formulated into recursive form. The improvement of the face recognition rate using this combination is quantified. The recognition rate by the combination is better than either appearance alone or depth alone. The performance of the algorithm is verified on both XM2VTS database and a real-time stereo vision system, showing that it is able to detect, track and recognize a person walking towards a stereo camera within reasonable time.