Recognizing human activities from temporal streams of sensory data observations is a very important task on a wide variety of applications in context recognition. Especially for timeseries sensory data, a method that takes into account the inherent sequential characteristics of the data is needed. Moreover, activities are hierarchical in nature, in as much that complex activities can be decomposed to a number of simpler ones. In this paper, we propose a two-stage continuous hidden Markov model (CHMM) approach for the task of activity recognition using accelerometer and gyroscope sensory data gathered from a smartphone. The proposed method consists of first-level CHMMs for coarse classification, which separates stationary and moving activities, and second-level CHMMs for fine classification, which classifies the data into their corresponding activity classes. Random Forests (RF) variable importance measures are exploited to determine the optimal feature subsets for both coarse and fine classification. Experiments show that with the use of a significantly reduced number of features, the proposed method shows competitive performance in comparison to other classification algorithms, achieving an over-all accuracy of 91.76%.