The human face is one of the most important objects in image or video. Color-based face detection techniques begin with skin color modeling. Color is usually the first cue sought for location face candidates in image or video because skin color is distinct. However, the human skin color differences among people lie in the intensity rather than the color itself. In this paper, we propose the new color modeling algorithm that can detect faces with various skin tones and the illumination conditions using TSK fuzzy system. We first develop a powerful skin and hair color detector based on the TSK fuzzy system considering intensity information in the HS color space. After locating skin pixel candidates, non-face blobs can be eliminated by using texture, shape or motion cues. However, it was computationally expensive due to the complicated algorithm. Hence, we employ the simple decision algorithm using the probabilistic convex regional relationship.