The sliding window method is a common approach for object detection. However, in order to detect relatively small objects in a large-size image, it can be substantially inefficient and require a huge amount of computation. While image downsizing or reduction techniques can be applied to resolve the drawbacks, they have high possibilities of losing essential information on small objects. To circumvent these problems for object detection, we propose an efficient hierarchical semantic grouping algorithm which consists of two parts: 1) Groping and 2) Recognition. The grouping part is to merge fragments using the similarity based on color and HOG features. Then, the recognition part is carried out based on the texton histogram model. In both parts, we use two types of rectangular patches from each fragment. We evaluated the proposed approach in comparison with other object detection methods, and then verified the outperformance and effectiveness of the proposed approach.