Estimating planar projective transform (homography) from a pair of images is a classical problem in computer vision. In this paper, we propose a novel algorithm for direct registering two point sets in using projective transform without using intensity values. In this very general context, there is no easily established correspondences that can be used to estimate the projective transform, and most of the existing techniques become either inadequate or inappropriate. While the planar projective transforms form an eight-dimensional Lie group, we show that for registering 2D point sets, the search space for the homographies can be effectively reduced to a three-dimensional space. To further improve on the running time without significantly reducing the accuracy of the registration, we propose a matching cost function constructed using local polynomial moments of the point sets and a coarse to fine approach. The resulting registration algorithm has linear time complexity with respect to the number of input points. We have validated the algorithm using points sets collected from real images. Preliminary experimental results are encouraging and they show that the proposed method is both efficient and accurate.