Recently, kernel-based tracking algorithms such as the mean shift tracking algorithm has been proposed, which use the information of color histogram together with some spatial information provided by the kernel. However, in spite of the fast speed, there exists an inherent instability problem which is due to the use of an isotropic kernel for spatiality and the use of the Bhattacharyya coefficient as the similarity function. In this paper, we will analyze how the use of the kernel and the Bhattacharyya coefficient can arouse the instability problem. Based on the analysis, we propose a tracking scheme that uses a new representation of the location of the target which is constrained by the color, the area, and the spatiality information of the target in a more stable way than the mean shift algorithm. With this representation, the target localization in the next frame can be achieved by a direct one step computation, and the tracking becomes stable, even in difficult situations such as low-rate-frame environment, and partial occlusion.