Despite significant progress in pedestrian detection has been made in recent years, detecting pedestrians in crowded scenes remains a challenging problem. In this paper, we propose to use visual contexts based on scale and occlusion cues from detections at proximity to better detect pedestrians for surveillance applications. Specifically, we first apply detectors based on full body and parts to generate initial detections. Scale prior at each image location is estimated using the cues provided by neighboring detections, and the confidence score of each detection is refined according to its consistency with the estimated scale prior. Local occlusion analysis is exploited in refining detection confidence scores which facilitates the final detection cluster based Non-Maximum Suppression. Experimental results on benchmark data sets show that the proposed algorithm performs favorably against the state-of-the-art methods.