Multiple sensor-based context inference systems can perceive users' tasks in detail while it requires complicated recognition models with larger resources. Such limitations make the systems difficult to be used for the mobile environment where the context-awareness would be most needed. In order to design and operate the complex models efficiently, this paper proposes an evolutionary process for generating the context models and a selective inference method. Dynamic Bayesian networks are employed as the context models to cope with the uncertain and noisy time-series sensor data, where the operations are managed by using the semantic network which describes the hierarchical and semantic relations of the contexts. The proposed method was validated on a wearable system with variable sensors including accelerometers, gyroscopes, physiological sensors, and data gloves.