According to the widespread mobile devices and wearable devices, various mobile video applications are emerging. Some of those applications contain sensitive data such as military information so that they need to be protected from anonymous intruders. Thus, several video encryption techniques have been proposed. Accordingly, it has become essential to evaluate the visual security of encrypted videos. Several techniques have attempted to evaluate the visual security in the spatial domain but failed to capture it in the temporal domain. Thus, we present a temporal visual security metric and consequently propose a spatio-temporal visual security metric by combining ours with an existing metric which evaluates the spatial visual security. Our experimental results demonstrate that our proposed metrics appropriately evaluate temporal distortion as well as spatial distortion of encrypted videos while ensuring high correlation with subjective evaluation scores. Further we examine the tradeoff between the energy consumption for mobile video encryption techniques and visual security of encrypted videos. This tradeoff study is useful in determining a right encryption technique which satisfies the energy budget for secure mobile video applications.