Construction is generally conducted in highly changeable site conditions due to operation of machinery, transfer of materials, moving workers, and changing progress status. Such dynamic characteristics can always lend themselves to the potential for safety accidents. However, it is not easy to have a generalized safety management system that can universally apply to every job site. Although there are some guidelines for safety management, their application to a construction site has to be tailored to satisfy the unique nature of the particular project. Based on the good understanding of the particular site, effective and appropriate safety management process should be derived for workers to be protected from potential dangers of the site. This paper presents an on-site safety management methodology based on image processing and fuzzy inference. Image processing is used to extract spatial information of workers; fuzzy inference is then used to provide the workers a proper level of safety warning based on the spatial information. Contextual site information, such as equipment operations and density of workers, also constitutes the on-site safety management methodology. The proposed methodology is expected to provide a safer working environment for construction workers through its capability to easily customize the safety management system for a construction site.