Activity recognition with mobile sensors is a challenging task due to the inherent noisy nature of the input data and resource limitations of the target platform. This paper presents a novel method of hybridizing classifier selection and classifier fusion in order to address these difficulties. It efficiently decreases the computational cost by activating appropriate classifiers according to the characteristics of the given input, and resolves the pattern variations by combining the chosen classifiers with localized templates. The proposed method is integrated with a wearable system that includes five motion sensors (accelerometers and gyroscopes), a set of bio-signal sensors, and data-gloves. The experiments on two different levels of activities, such as 11 primitive motions and eight composite behaviors, demonstrated that the proposed method is useful to the wearable systems.