Gesture recognition provides a new interface to user. Various methods for the gesture recognition are feasible in smartphone environment since a number of sensors attached are gradually increasing. In this paper, we propose a gesture recognition method using smartphone accelerometer sensors. The high false-positive rate is definite if the gesture sequence data are increased. We have modified BLSTM (Bidirectional Long Short-Term Memory) recurrent neural network with non-gesture rejection model to deal with the problem. A BLSTM model classifies the input into the gesture and non-gesture classes, and the specific BLSTM models for the gestures further classify it into one of twenty gestures. 24,850 sequence data are used for the experiment, and it consists of 11,885 gesture sequences and 12,965 non-gesture sequences. The proposed method shows higher accuracy than the standard BLSTM.