This paper proposes a framework of segment-based free space estimation using plane normal vector with stereo vision. An image is divided into compact superpixels and each of them is viewed as a plane composed of the normal vector in disparity space. To deal with the variation of illumination and shading in real traffic scenes, we estimate depth information for the segmented stereo pair. The representative normal vector is then computed at superpixel-level, which alleviates the problems of conventional color-based approaches and depth-based approaches simultaneously. Based on the assumption that the central-bottom of input image is navigable region, the free space is then determined by clustering the plane normal vectors with the K-means algorithm. In experiments, the proposed approach is evaluated on the KITTI dataset in which we provide the ground truth labels for free space region. The experimental results demonstrate that the proposed framework effectively estimates the free space under various real traffic scenes, and outperforms current state of the art methods both qualitatively and quantitatively.