Nowadays, lots of people concern about the privacy violation in the video surveillance systems widely utilized in order to protect people and their properties. Various protection techniques have been proposed to protect the privacy-sensitive information in the video surveillance and they properly provide the protection. Unfortunately, the video compression rate and the energy consumption have not been thoroughly investigated in the video surveillance system providing the privacy protection. However, the advance from the wired system to the wireless system which has the limited resources makes the video compression rate and the energy consumption as important as the degree of the perceptual protection and the recognition accuracy in the recovered video. In this paper, we propose the configurable privacy management technique in order to enhance both the compression rate and the energy efficiency of the privacy-protected video surveillance system. Satisfying the highest 10% degree of the perceptual protection in our experiments, the proposed method reduces the compressed bitstream size and the energy consumption by up to 66.0% and 27.3% each as compared to no protection, that is, the compression only while achieving the fine recognition accuracy by 82.1% in the recovered video.