This paper presents a vision-based vehicle detection system in the infrared (IR) and vision system using an effective feature extraction and algorithm. This system follows two steps: Hypothesis Generation (HG) method and Hypothesis Verification (HV) method. In HG method, vertical and horizontal edges are used. To extract these edges effectively a neighborhood gradient prediction(NGP) edge detection is used. With these extracted edges, the vehicle location candidates are generated. This step reduces the computational time in comparison with exhaustive search method. In HV method, the effective feature extraction such as HOG and GABOR feature are used. A support vector machine (SVM) for classification is also used. This step verifies if the vehicle candidates are vehicle or not. The test image is obtained by a monocular IR camera and visible camera attached on moving vehicle. In vision image, NGP method is compared with SOBEL method. In IR image, HOG feature is compared with GABOR feature.