As the sensors embedded to a smartphone are proliferating, many application systems for context-aware services are actively investigated. This paper proposes a gesture recognition system with smartphones for better interface. It is important to maintain high accuracy even with the large number of gestures. To improve the accuracy, we adopt the recurrent neural network based on hierarchical BLSTM (Bidirectional Long Short-Term Memory). The first level BLSTMs are used to discriminate the gestures and nongestures, and the second level BLSTMs classify the input into one of twenty gestures. Experiments with 24,850 sequence data consisting of 11,885 gesture sequences and 12,965 non-gesture sequences confirm the high performance of the proposed method over the competitive alternatives.