Various video applications in mobile and wearable devices deal with private or important video data. In order to protect the important video information, several video encryption techniques have been proposed. The secure video processing, the combination of the video compression/decompression and the video encryption/decryption, causes lots of computational overheads, thereby consuming huge energy. Therefore, mobile and wearable devices which have limited battery capacity need to efficiently manage the power resources when performing the secure video processing. In order to find the configuration of the secure video processing which satisfies both the energy and visual security constraints, examining the tradeoff space in the secure video processing is significantly essential. In this paper, we study the tradeoff space between the energy consumption and the visual security in the secure video processing. Consequently, we propose the EVS (Energy-bound Visual Security) to find the interesting configuration of the secure video processing in order to achieve the maximum visual security under the energy budget, by exploring the huge tradeoff space. Moreover, we design an exploration technique to reduce the overhead of investigating the tradeoff space. Thanks to the EVS, we improve the visual security by up to about 17% under the same energy budget in our experiments. In addition, our proposed technique reduces about 55% exploration overhead in the experiments.